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Summary 

A pilot’s ability to acquire approach information 
efficiently is tied closely to the safe, efficient execution 
of approach and landing procedures. Currently, most of 
the information that is needed for the preapproach and 
approach segments is presented on approach charts. The 
procedures and information requirements of the approach 
segments are arguably more complex than those that 
comprise the other flight phases, and being able to ade- 
quately present all necessary information presents a chal- 
lenge for cartographers and flight deck system designers. 

Many tasks associated with using approach charts 
are cognitive (e.g., planning the approach and monitoring 
its progress). Mental processes that are used to manipu- 
late information in support of higher level cognitive 
tasks, such as categorization and prioritization, are inte- 
gral to efficient execution of the higher level tasks. A 
successful system interface is one that conforms to the 
user’s mental models. Matching the presentation formats 
of approach information with the mental processes used 
by the pilots to manipulate the information should 
improve the pilots’ efficiency when they are executing 
the higher level cognitive tasks. A new methodology was 
developed for this study to provide designers with data 
that are necessary to make such design decisions. The 
methodology used in this study enhances traditional 
information requirements analyses by combining psy- 
chometric scaling techniques with a simulation task to 
provide quantifiable links between pilots’ cognitive rep- 
resentations of approach information and their use of 
approach information. 

The objectives of this study were to determine 
1) how pilots categorize and prioritize approach informa- 
tion, 2) how pilots acquire approach information, and 
3) the relationship between these cognitive models (i.e., 
categorization and prioritization) and how pilots use the 
information. The results indicated that pilots mentally 
organize approach chart information into 1 0 primary cat- 
egories: Communications , Geography , Validation , Ob- 
structions, Navigation , Missed Approach , Final Items , 
Other Runways , Visibility Requirement , and Navigation 
Aids. These similarity categories were found to underlie 
other mental models and many of the higher level cogni- 
tive processes used by pilots to accomplish their tasks. 
For instance, items within a given category were of simi- 
lar importance in the average rankings provided by the 
pilots. Additionally, when pilots were tested on what 
approach information they could recall from their last 
approach, the category to which an information element 


belonged was found to be a good indication of whether or 
not the information would be remembered, independent 
of the number of items that comprised the category. 

When pilots were flying approaches, acquisitions of 
information elements were found to be highly related to 
the categories to which the elements belonged. Certain 
categories of information were more likely to be selected 
than others, regardless of the number of items that com- 
prised the category. In general pilots were more likely to 
select their next information element from the category 
they were already sampling, rather than change to a new 
category. Since it is reasonable to assume that pilots will 
select information that is functionally related to a task, 
these transition results provided added evidence that the 
identified categories contain similar information and 
insight into how the information might be tailored if pre- 
sented electronically. An analysis of variance (ANOVA) 
that was conducted on acquisitions from these similarity 
categories found statistically significant main effects of 
crew member (pilot flying versus pilot not flying), preci- 
sion (instrument landing system (ILS) versus very 
high frequency (VHF) omnidirectional range (VOR)), 
weather (good versus adverse), and flight phase 
(preapproach versus approach). Another finding was 
that the requirements (reflected by the acquisitions) 
changed as a function of approach segments. This find- 
ing suggests that the categories can be used to help define 
both the presentation formats (e.g., salience and repeti- 
tion) and the tailoring of information (when presented 
electronically). 

Results of this study augment previous information 
requirements analyses by identifying what information is 
acquired, when it is acquired, and which presentation 
concepts might facilitate its efficient use by better match- 
ing the pilots’ cognitive model of the information. Since 
the similarity categories were salient in the pilots’ use of 
approach charts, using these underlying categories of 
information in designing presentation formats may ease 
related pilot workload by not requiring the pilot to orga- 
nize the information to the same extent prior to using it. 

Introduction 

An instrument approach chart provides information 
necessary to descend safely from an en route altitude on a 
feeder route (usually following a standard arrival proce- 
dure) to the landing altitude of a destination runway. The 
ability of a pilot to acquire approach information 
proficiently is tied closely to the safe, efficient execu- 
tion of approach and landing procedures. Most of the 



information that is needed for the approach segments 
(transition, intermediate, final, and missed) 1 is presented 
on the approach chart. The procedures and informa- 
tion requirements of these segments are arguably the 
most complex, and presenting all the needed informa- 
tion (e.g., frequencies, headings, and altitudes) on a 
5 1/2 in. x 8 1/2 in. approach chart is challenging for the 
designer. There are currently two primary approach chart 
formats published today, one by Jeppesen Sanderson, 
Inc., and one by the National Oceanic and Atmospheric 
Administration (NOAA) National Ocean Service (NOS) 
(appendix A). 

Both approach chart formats are products of years of 
evolutionary changes. Appropriate changes were made to 
the charts as new information and format requirements 
were identified. Many of these changes were either rec- 
ommended by pilots or derived from information 
requirements analyses for the approach and landing 
phases of flight. Previous information requirements anal- 
yses examined information that is necessary for observ- 
able descent and landing tasks (Boeing 1991a, 1991b; 
Hansman and Mykityshyn 1990; Mykityshyn and 
Hansman 1990, 1991). They did not account explicitly 
for more unobservable cognitive tasks such as planning 
and monitoring or for the requirements of the cognitive 
processes that support these tasks. 2 For example, how the 
pilots categorize and prioritize information are two 
salient cognitive processes (models) that support higher 
level cognitive tasks and are not explicitly addressed 
with traditional analyses (Jonsson and Ricks 1995). Such 
omissions in the design of presentation formats may limit 
pilot efficiencies since most of the tasks involved with 
using an approach chart are cognitive. 

Cognitive processing refers to mentally representing, 
organizing, and manipulating information. These mental 
representations are often referred to as cognitive models. 
Such models of a person’s environment serve several 
important functions, not the least of which is that they 
speed the user’s rate of comprehension by allowing situa- 
tions, objects, functions, and relationships to be classi- 
fied by important or salient features (Cannon-Bowers, 
Tannenbaum, and Salas 1991). As stated previously, 

Ut A feeder or terminal route may be used to take you from the 
en route structure to an initial approach fix (IAF). Next, you follow 
an initial approach segment to an intermediate fix (IF). From here, 
you follow the intermediate segment to the final approach fix 
(FAF). The final approach segment ends at the runway, airport, or 
missed approach point (MAP). Upon reaching the MAP, if you are 
unable to continue the approach to a landing, you follow the missed 
approach segment back to the en route structure.” (Jeppesen 1993) 

2 

Previous studies applied analytical methods to investigate cog- 
nitive demands of approach chart use; therefore, they did not pro- 
vide an empirical (i.e., explicit) connection. 


these models are integral to the efficient execution of the 
cognitive tasks associated with approach chart manage- 
ment and are dominant in most tasks that involve infor- 
mation retrieval. For instance, among the information 
retrieval tasks pilots perform during the approach and 
landing phases of flight are those used to acquire infor- 
mation to build and validate their mental picture of the 
approach and landing. 

Much of the literature in the area of cognitive models 
suggests that many performance advantages are realized 
when information is presented to the users in formats that 
conform to their cognitive models (Clay 1993). For 
example, collocating items that the user perceives as sim- 
ilar will improve access time and retention. The converse 
is also thought to be true: not matching the user’s mental 
model of the information in a presentation format will 
decrease the efficiency of its use (e.g., adding to the time 
required to find items). Clay (1993) states that many of 
the errors associated with presentation formats are a 
result of discrepancies between the designers’ model of 
the system and the users’ mental model of the system. 

There is an opportunity to enhance the presentation 
of approach information by examining the cognitive 
demands of approach chart use in conjunction with new 
design efforts. There are currently efforts underway to 
improve the presentation formats of information on 
approach charts and to establish more consistency 
between the two existing formats. There are also research 
efforts that explore formats for electronically presenting 
approach information on future flight decks. Given the 
efficiency advantages of matching the supply of informa- 
tion to the user’s expectation, it would be beneficial for 
designers associated with presenting approach and land- 
ing information to design their formats to conform to the 
pilots’ cognitive models. 

As mentioned earlier, traditional information re- 
quirements analyses are limited because they do not 
account explicitly for cognitive demands and therefore 
do not supply the designer with information that is perti- 
nent to the format decisions based on cognitive require- 
ments. Most traditional analyses of the “task 
decomposition” variety are very good at identifying 
observable tasks (e.g., descent) while not accounting for 
cognitive tasks (e.g., planning). Given the advantages 
associated with matching the presentation of information 
to the user’s mental model of the information, new tech- 
niques were developed for this study to enhance tradi- 
tional information requirements analyses by supplying 
quantifiable links between the pilots’ cognitive models of 
approach information and their methods of acquiring it. 

The premise of the experimental method described 
in this paper is that traditional information requirements 
analyses could be enhanced greatly by combining the 
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results obtained from psychometric tasks (explicitly 
designed to examine cognitive processing) with those 
obtained by using a nonintrusive technique that examines 
approach chart information acquisitions in a simulator 
environment. Psychometric scaling tasks are usually 
done in laboratory environments in which subjects 
manipulate information on the basis of its significance in 
the real environment. Scaling this type of data provides 
quantifiable descriptions of people’s cognitive represen- 
tations (e.g., similarity categories). The new techniques 
that were developed for this study combined these tradi- 
tional laboratory tasks with a new domain-specific data 
acquisition technique in order to provide an empirical 
link between the cognitive models that result from the 
psychometric scaling and how the pilots use them. 

Central to this experiment was this new domain- 
specific data acquisition tool that the pilots used when 
they were flying simulated approaches. The tool 
recorded what approach chart information the pilots 
acquired and when they acquired it. This objective was 
accomplished by providing the flight crew with approach 
chart information on computer screens that were driven 
by a program that requires them to acquire the informa- 
tion actively. Data that were obtained with this acquisi- 
tion tool and the data from the psychometric scaling tasks 
provided quantifiable links between how the pilots cog- 
nitively model approach information and how and when 
they use it. 

Objectives 

There were three primary objectives of the research 
reported in this paper. The first was to identify how pilots 
categorize and prioritize approach chart information. The 
second objective was to determine how pilots acquire the 
information when they are flying instrument approaches. 
This objective prompted the design of a new acquisition 
technique which recorded data on the information that 
pilots acquired and when they acquired it during each 
phase of the approach. The third objective was to assess 
the relationship of the pilots’ cognitive models and the 
manner in which the pilots acquire approach chart infor- 
mation. Issues associated with this objective were 

1 ) determining the correlation between the pilots’ cate- 
gorization models and their acquisition methods, 

2) assessing the relationship between the pilots’ prioriti- 
zation models and their acquisition patterns and frequen- 
cies, and 3) determining the relationship between their 
recall of information and acquisition practices. 

Abbreviations 

AGL above ground level 

ANOVA analysis of variance 


ATC 

air traffic control 

ATIS 

Automatic Terminal Information Service 

DC 

Douglas Commercial 

DFW 

Dallas-Fort Worth 

DME 

distance measuring equipment 

ELS 

electronic library system 

FAF 

final approach fix 

GS 

glideslope 

IAF 

initial approach fix 

IF 

intermediate fix 

ILS 

instrument landing system 

MAP 

missed approach point 

MAPLIST 

managing approach plate information study 

MDPREF 

multidimensional preference 

MDS 

multidimensional scaling 

MM 

middle marker 

MSA 

minimum safe altitude 

MSL 

mean sea level 

NASA 

National Aeronautics and Space 
Administration 

NAV 

navigation display 

NDB 

nondirectional beacon 

NOAA 

National Oceanic and Atmospheric 
Administration 

NOS 

National Ocean Service 

ORD 

Chicago-O’Hare 

PF 

pilot flying 

PCPREF 

persona] computer (multidimensional) 
preference 

PFD 

primary flight display 

PNF 

pilot not flying 

TDZE 

touchdown zone elevation 

VHF 

very high frequency 

VOR 

VHF omnidirectional range 

VSI 

vertical speed indicator 


Method 

The following sections describe an experiment 
entitled Managing Approach Plate Information Study 
(MAPLIST) that was conducted at Langley Research 
Center. As a precursor to the experiment that is reported 
in this paper, an exploratory study was conducted that 
concentrated primarily on the feasibility of the methodol- 
ogy. Results from this preliminary study were not 
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published and will be referred to in this paper as the 
exploratory study. From findings in the exploratory 
study, MAPLIST was refined and formally conducted 
from September 1993 through January 1994 (including 
piloted checkout sessions). Each crew’s testing required 
approximately 7 hr and was completed in 1 day. 

Subjects 

Participants in this study were tested as members of 
a flight crew. For background training consistency, each 
flight crew came from the same airline and had a cur- 
rently active captain and a first officer (i.e., no captains 
performed the role of first officer or vice versa). All par- 
ticipants were currently flying commercially. Data were 
collected from 13 DC-9 (aircraft restriction based on 
simulator used) flight crews who came from American 
Airlines, Express One International, and USAir. The 
average age of pilots participating in the study was 
41 years. The youngest pilot was 29 and the oldest was 
53. Average commercial flying time for each pilot was 
8786 hr, with a minimum of 3500 hr and a maximum of 
20000 hr. 

Stimuli 

For the psychometric scaling tasks described below, 
pilots were required to work with the types of informa- 
tion contained on an approach chart. The information 
element types were compiled in a list (table 1) using the 
Jeppesen Sanderson, Inc., Instrument Rating Manual 
(Jeppesen 1993). Added to this list was some information 
pertaining to runway data (typically found on a separate 
chart with the Jeppesen format and on the same chart 
with the NOS format. (See appendix A.) 

Included in table 1 are two element types that were 
inadvertently omitted from the psychometric scaling 
tasks: middle marker (MM) crossing altitude above 
ground level (AGL) and MM crossing altitude mean sea 
level (MSL). These two element types were not part of 
the list that was used to generate the stimuli but were 
present on the charts that were used during the simulation 
portion and are therefore included in table 1. Also 
included in table 1 are the numbers of corresponding 
approach chart items (of each type) that were used in this 
study. 3 

Approach charts were needed for the simulation por- 
tion of the experiment. From observations in the explor- 
atory study, difficulty of the approach procedure only 
affected how often a subject acquired information. More 

3 Since the procedure turn was not part of any of the flight profiles 
used in this study, the procedure turn altitude depicted on the charts 
(in the profile section) was counted as a step-down altitude and ex- 
plains the zero item account for procedure turn altitude in table 1. 


acquisitions were made during the preapproach segment 
and fewer were made in the flying segments of difficult 
approaches. Difficulty as a variable did not affect what 
information was acquired; therefore, it was decided not 
to vary the approach difficulty in the study that is 
reported in this paper. 

Four separate approach charts were developed for 
use in the experimental conditions. To maintain the same 
level of approach difficulty across the four charts, each 
chart was a transformation of McCarren International in 
Las Vegas, Nevada, instrument landing system (ILS) 
25R and VOR 25L/R. (The McCarren International pro- 
cedures chosen for this study were deemed neither diffi- 
cult nor easy.) The chart transformations involved 
changes to all names (e.g., fix names and navigation aid 
names), frequencies, altitudes, and rotations to the basic 
geography of the airport. These transformations served to 
eliminate the possible impact of approach familiarization 
(e.g., memory) on the acquisition of information that 
might have occurred with the use of an existing chart or 
the same chart. The four approach charts used for data 
acquisition in this study can be seen in figures 1 
through 4. 

For this study, the approaches were divided into two 
major phases: preapproach and approach. Preapproach 
refers to the time preceding the flying portion of the 
experiment. During preapproach, the airplane was sus- 
pended on a feeder radial at top of descent (approxi- 
mately 20000 feet AGL) 50 n.mi. from the initial 
approach fix (IAF). The approach phase was further 
divided into four segments: transition , intermediate , 
final, and missed. In this study transition refers to the 
segment starting at top of descent and ending at the IAF. 
Intermediate refers to the segment starting at the IAF and 
ending at the final approach fix (FAF). Final approach 
was the segment starting at the FAF and ending at the 
runway threshold or when initiating missed approach 
procedures. Missed approach refers to the period at 
which the missed approach procedures are first initiated. 

The Jeppesen format is most representative of the 
charts used in this study. Given the task in which the 
pilots used these charts (described below), it was impor- 
tant that the subjects be familiar with where and 
how information was presented. Since all the major 
United States commercial airlines and 90 percent of all 
airlines worldwide use Jeppesen charts (Mykityshyn and 
Hansman 1991) and because the subjects were active air- 
line pilots, this format best suited the study’s objectives. 
As can be seen in figures 1 through 4, airport information 
was added to the bottom right comer (i.e., as in the NOS 
formats), which allowed the presentation of desired 
airport information without providing the pilots with 
additional charts. It is important to understand that this 


4 



Table 1. Element Types and Number of Chart Items 


Information element 

Items 

Information element 

Items 

Information element 

Items 

Airport departure freq. 

8 

Geographic name 

4 

Notes 

2 

Airport elevation 

12 

Ground freq. 

mm 

Obstruction 

70 

Airport name 

7 

Highest reference point 

4 

Other runway numbers 

43 

Approach freq. 

4 

Holding pattern course 

8 

Outbound course 

6 

ATIS arrival freq. a 

4 

ILS identifier 

mm\ 

Primary facility freq. 

12 

ATIS clearance freq. b 

4 

ILS localizer freq. 

2 

Primary fac. Morse code 

4 

ATIS departure freq. 

4 

ILS localizer mag. course 

2 

Procedure name 

20 

Chart date 

4 

ILS Morse code 

2 

Procedure turn alt. 

0 

Chart index number 

4 

Inbound course 

16 

Procedure turn dist. limit 

2 

Changes 

0 

Lat-long grid 

19 

Rate of descent 

12 

Decision alt. (AGL) 

26 

Lat-long of airport VOR 

4 

Runway length 

8 

Decision alt. (MSL) 

26 

Lead radial 

10 

Scaling 

0 

Distance (non-DME) 

34 

Location of MAP 

8 

Step-down alt. (AGL) C 

8 

DME distance 

30 

Magnetic variation 

4 

Step-down alt. (MSL) 

8 

Effective date 

4 




4 

FAF intercept alt. (AGL) 

4 

MAP (nonprecision) 

8 

TDZE target runway 

8 

FAF intercept alt. (MSL) 

4 




2 

Feeder route radial 

14 

Missed approach instr. 

| 

Time from FAF to MAP 

24 

Fix name 

22 


2 

Tower freq. 

4 

Fix name at FAF 

16 


2 

Visual descent point 

0 

Fix name at IAF 

5 

MSA minimum alt. c 

16 

Visibility requirement 

52 

Glideslope angle 

2 

MSA reference point 

4 


11 

GS intercept alt. (AGL) 

2 

MSA sector radials 

16 

VOR identifier 

28 

GS intercept alt. (MSL) 

2 

NDB freq. 

0 

VOR Morse code 

8 

GS pos. on parallel runway 

0 

NDB identifier 

0 


5 

GS pos. on target runway 

2 

NDB Morse code 

0 




a Most airports have only one Automatic Terminal Information Service (ATIS) frequency for both arrival and departures. 
Some airports have two ATIS frequencies (arrival and departure) to accommodate the use of different runways for approach 
and landing procedures because the active runway is commonly given in the ATIS announcement. 

b The clearance delivery frequency (provided in the same section of the airport diagram as the departure ATIS) was inad- 
vertently entitled ATIS clearance frequency. None of the pilots expressed confusion concerning this oversight or difficulty in 
interpreting its meaning. 

c Step-down altitudes are sometimes referred to as minimum crossing altitudes at step-down fixes. 
d These elements were not part of the psychometric scaling data but were items on the charts that were used. 
e Minimum safe altitude (MSA) appears in the table as MSA minimum alt. as it did on the stimulus card. The redundancy 
in MSA minimum alt. was not intentional and led to no apparent difficulties with regard to the pilots’ tasks. The stimulus 
should have read simply MSA. 


format is not a recommendation or an endorsement of 
any specific presentation format. This study was con- 
cerned exclusively with information requirements analy- 
ses, not with presentation concepts. 

Equipment 

For the information acquisition portion of the experi- 
ment, flight crews flew scenarios in the specially config- 
ured Langley DC-9 simulator. The Langley DC-9 
simulator was a full workload, fixed-base, series-30 
cockpit simulator. Two visual displays provided out-the- 
window scenes for each crew member. Full workload 


studies were supported by this simulator since the capac- 
ity existed to simulate all aircraft instruments, annuncia- 
tors, switches, and alarms. The autopilot was functional 
for this experiment, and subjects were encouraged to use 
it as they would when flying for their airline. The config- 
uration of the Langley DC-9 simulator for this experi- 
ment can be seen in figure 5. 

Air traffic control (ATC) was simulated at a work- 
station in a room adjacent to the Langley DC-9 simula- 
tor. Three different ATC controllers (only one per 
session) fulfilled this responsibility. At the controller’s 
station was a computer screen that indicated the aircraft’s 
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position on the current approach; instrument readings 
that indicated the altitude, airspeed, and distance from 
the runway threshold; and scripts for each scenario that 
indicated key aspects to ensure test consistency. Control- 
lers also were able to view actions within the cockpit by 
monitoring the videotaping. The primary responsibility 
of the controllers was to maintain the time profile consis- 
tency of each flight crew by issuing realistic ATC 
instructions (e.g., speed reductions). Figure 6 shows the 
ATC station for MAPLIST. 

The flight crews’ computer interfaces to the acquisi- 
tion task were located outside the simulator windows to 
the left of the captain and to the right of the first officer 
(fig. 5). The two computers operated independently, 
which means that computer interactions by one pilot 
were not apparent on the screen of the other pilot. For the 
acquisition task, a grayed-out (i.e., items made illegible) 
form of the approach chart required for the current sce- 
nario was generated on the screen (fig. 7). With the cur- 
sor control device, the pilot could point to and select an 
item by pressing on (i.e., clicking) the cursor control 
device button. 

Once selected, an item would stay legible for 15 sec- 
onds 4 before returning to its illegible (i.e., grayed-out) 
state. An item could be selected as many times as 
desired, and more than one item could be made legible at 
one time by simply clicking on multiple items within 
15 seconds of one another. Figure 8 shows a grayed-out 
chart and the same chart with some items selected. 

Subjects were given access to each approach chart 
(on the acquisition device) prior to engaging in the flying 
portion of each scenario. During this preapproach phase, 
the pilots were given their initial position, heading, alti- 
tude, and ATTS information. Pilots were instructed to 
become familiar with the approach and the captain was to 
do a formal preapproach briefing. The captain was 
always the pilot flying (PF) and the first officer was the 
pilot not flying (PNF) for all data runs. 

Recording of the information selected prior to the 
flying segment (referred to as preapproach in this study) 
was marked for differentiation from that selected during 
the flying portion to assist in post analysis. The informa- 
tion items selected and the time of selection were 
recorded separately for each pilot. In addition to the 
number of selections and elapsed time, segments of the 
approach (i.e., transition, intermediate, final, and missed) 


4 Fifteen seconds was a duration derived from feedback and 
observation during the exploratory study and the checkout portion 
of the study described in this paper. Fifteen seconds afforded 
enough time to gather the information and combine it with the pre- 
sentation of other information and also required the pilots to acquire 
information explicitly again if they needed it later. 


were marked in the data. As stated previously, a major 
function of ATC in this study was to give instructions to 
the crews that would help maintain consistent times 
within the flight segments between flight crews. 

Procedure 

As mentioned above, MAPLIST combined psycho- 
metric scaling tasks with a domain-specific information 
acquisition task in order to assess how pilots cognitively 
model approach chart information, how they acquire the 
information, and how the cognitive models relate to the 
acquisition of this information. Pilots were first required 
to complete a background questionnaire regarding their 
experience and age. They were then given a brief 
description of the study, their schedules for the day, and 
the tasks they would be performing. The tasks were com- 
pleted in 1 day and are described below in the order in 
which they were administered. 

Information Sorting 

The first task that was accomplished by each subject 
was a card-sorting task. Subjects were given a deck of 
index cards, each containing 1 of the 75 information ele- 
ments listed in table 1 . Note that the 75 elements do not 
include middle marker (MM) crossing altitude (AGL) or 
MM crossing altitude (MSL). Instructions were given to 
sort the elements into piles according to similarity. The 
subjects were instructed to build as many similarity piles 
of cards as they felt were necessary and were told that the 
number of cards in each pile could vary. No definition of 
similarity was supplied to the subjects. After the cards 
were sorted, the number of groups and the cards within 
each group were recorded. 

Information Ranking 

Following the similarity sorting, each subject was 
instructed to rank the same 75 information elements from 
1 (most important) to 75 (least important) according to 
the pilot’s perception of the information’s importance. 
No context or definition of importance was given to the 
pilots. After the cards were ordered by the pilot, the posi- 
tions were recorded. Immediately following this context- 
free ranking, the subjects were then instructed to rank the 
importance of the elements when they were flying an 
approach where both visibility and cloud bottom are at 
minimums (called adverse weather condition). The posi- 
tions of each card were again recorded. 

Information Acquisition 

Flight crews participated in an acquisition task 
which took place in the Langley full-workload DC-9 
simulator (described previously). Each crew was given 
a briefing and training concerning the simulator, its 
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equipment, and the acquisition task. Each crew’s certifi- 
cation for the DC-9 airplane was current, which mini- 
mized the time required for simulator and cockpit 
equipment familiarization. 

Familiarity runs . In order to make the pilots familiar 
with the simulator, the approach chart display, and the 
means for acquiring approach chart information, each 
crew was required to fly two preliminary approaches 
before starting the set of scheduled data runs. The two 
approaches used for practice were the same for each 
crew, and to avoid familiarity with specific approach 
information they were not used again during the data col- 
lection portion. Each approach was preceded with a pre- 
approach phase in which the crew was given the initial 
position, heading, speed, altitude (always level flight at 
approximately 20000 ft AGL), and ATIS information. 
Each approach phase was initiated by ATC instruction (a 
descend and maintain instruction). Flight crews were 
provided with paper and writing utensils for recording 
initial conditions and ATC instructions but were 
instructed not to record any of the information from the 
approach chart screen. 

Data acquisition runs . To test differences in acqui- 
sitions under certain factors, three independent variables 
were chosen. The independent factors in the simulation 
portion of the experiment included 1 ) approach precision 
(very high frequency omnidirectional range (VOR) ver- 
sus instrument landing system (ILS)), 2) weather (good 
versus adverse), 5 and 3) flight phase (preapproach versus 
approach). The treatment combinations were counter- 
balanced to control for possible effects caused by order 
(table 2). As noted above, each crew consisted of a cap- 
tain, who was always the PF, and a first officer, who was 
always the PNF. This design was analyzed by using a 


5 Good weather was defined as unrestricted visibility and a mea- 
sured ceiling well above the decision altitude (approximately 
2000 ft AGL). Bad weather was defined as 3/4 mile visibility (just 
legal) and a measured ceiling barely above the decision altitude. 


mixed-model analysis of variance (ANOVA) with 
PF-PNF (termed crew member) as the between-subject 
variable, while precision, weather, and flight phase were 
the within-subject variables. 

Again, the difficulty of the approaches was held con- 
stant, and to ensure that familiarity was not an issue, four 
approach charts (figs. 1 through 4) were created for this 
test design. The approach chart numbers correspond to 
the condition numbers (the numbers in parentheses) 
noted in table 2. 

As with the familiarity runs, each data run was pre- 
ceded with ATIS information, aircraft location, and time 
for the pilots to become familiar with the approach and to 
do a preapproach briefing. The time required for each 
scenario (preapproach plus approach) was approximately 
20 min. 

Information Recall 

Immediately after completing all scheduled simula- 
tion runs, each pilot was given a grayed-out paper copy 
of the chart that was used during the last approach. The 
pilots were instructed to do this task individually and to 
write on the paper copy all the information (e.g., specific 
courses, altitudes, and frequencies) that they could recall, 
whether or not they were completely confident regarding 
its accuracy or completeness. To prevent their changing 
the manner in which they would have acquired the infor- 
mation (e.g., extra acquisitions to improve retention), 
pilots were given no advance knowledge that they would 
be given a memory test. 

Retrospective Debriefing 

After the information recall test, each crew partici- 
pated in a debriefing session. During the debriefing, 
audio recordings were made of the pilots’ explanations 
for each information item that was selected during their 
last flights. Crews were questioned together for this task, 


Table 2. Simulation Test Design 


Crew condition 

Flown first 

Flown second 

Flown third 

Flown fourth 

Crew 1, Crew 5, 
Crew 9 

(1) Nonprecision 
adverse weather 

(2) Nonprecision 
good weather 

(3) Precision 

adverse weather 

(4) Precision 
good weather 

Crew 2, Crew 6, 
Crew 10 and 10b* 

(2) Nonprecision 
good weather 

(3) Precision 

adverse weather 

(4) Precision 
good weather 

(1) Nonprecision 
adverse weather 


(3) Precision 

adverse weather ! 

(4) Precision 
good weather 

(1) Nonprecision 
adverse weather 

(2) Nonprecision 
good weather 

Crew 4, Crew 8, 
Crew 12 

(4) Precision 
good weather 

(1) Nonprecision 
adverse weather 

(2) Nonprecision 
good weather 

(3) Precision 

adverse weather 


*Crew 10 runs were repeated by another crew because of abnormality in one condition that was flown (refer to 
“Results” section). 
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but the pilots were afforded the opportunity to explain 
their individual acquisitions. To assist in this re- 
enactment, a paper version of the approach chart and the 
listing of selected items were used by the experimenter to 
guide the pilots through the selections they had made. 
During this exchange pilots were encouraged to elaborate 
on why they had selected each piece of information. 

Questionnaire 

Subjects completed their day of testing by answering 
a five-part questionnaire. Questions pertained to the sim- 
ulator’s fidelity, use of the acquisition device, approach 
chart adequacy, strategies when using the information 
acquisition task, and the experimental conditions. 

Results and Discussion 

As stated previously, the objectives of this research 
were threefold. This section has been organized first by 
the research objectives and then by the analyses per- 
formed in support of them. The first section addresses the 
cognitive models that pilots have of approach chart infor- 
mation. The second section addresses the pilots’ acquisi- 
tion of information when they are flying instrument 
approaches. 6 The third section addresses the last objec- 
tive: the relationship between the pilots’ cognitive mod- 
els and the way they acquire the information. The fined 
section discusses possible implications of these results on 
the pilots’ information requirements of approach chart 
use. 

Cognitive Models of Approach Information 

Knowing how pilots categorize and prioritize 
approach information should provide key knowledge for 
the designers of presentation concepts. Having informa- 
tion presented to people in a way that matches their men- 
tal model has been shown to improve the efficiency of 
their task performance (Clay 1993). The analyses and 
results described in this first section address two of the 
more salient cognitive models of approach chart informa- 
tion that pilots have: categorization and prioritization. 
The data used for these analyses were obtained from the 
similarity sorting and ranking tests described previously. 

6 Crew 10 simulation data were eliminated from the analyses de- 

scribed in the “Results” section because of a procedural error that 
occurred when scenario 2 was being flown. Another crew, labeled 
Crew 10b, was added to replace Crew 10 simulation data. Eliminat- 
ing Crew 10 data for scenario 2 was based on an unacceptable 
z score for the total flight time of that scenario. Please refer to the 
tables in appendix B for time profiles and z scores of all the simu- 
lation test runs. Psychometric scaling data were retained from all 
26 subjects since the flying portion did not affect these data. 


Similarity of Approach Chart Information 

Elements 

Each of the 26 pilots sorted the approach information 
elements (into piles) based on their similarity. From 
these sortings, a single 75 x 75 matrix (i.e., the proximity 
matrix) was constructed where the columns and rows 
corresponded to the stimuli, and the number in each cell 
corresponded to how many pilots put the two stimuli in 
the same pile. This matrix was then used in a cluster 
analysis and in a multidimensional scaling analysis. 

Cluster analysis. Recent developments in cognitive 
research have demonstrated the usefulness of psychomet- 
ric techniques in representing human knowledge and 
information processing (Ashby 1992; Nosofsky 1984, 
1986; summarized in Nosofsky 1992). One such method, 
cluster analysis, identifies items that are closely associ- 
ated with one another, groups them, and provides a hier- 
archical representation of the stimuli, thereby allowing 
the investigator to examine the representation for obvi- 
ous or intuitive categories. Clusters of information tell 
researchers how pilots define category membership. 

A cluster analysis was done on the similarity data 
with SYSTAT (see Wilkinson, Hill, and Vang 1992) 
using the average linkage method. The number of clus- 
ters selected for interpretation was determined by the 
clustering solution that exhibited the highest level of sta- 
bility across the widest range of the Euclidean distance 
metric (Romesburg 1984). In other words, the cut was 
made at a point at which moving it within a wide range 
did not affect the number of clusters. 

Three evaluators independently examined the clus- 
tering solution for the point (i.e., cut position) where the 
number of clusters was constant for the greatest distance. 
There was unanimous agreement that the cut could be 
made at either the 7- or 10-cluster level. Qualitatively, 
the evaluators felt that the 10-cluster solution would 
yield a better interpretation, and it was selected. Figure 9 
shows the clustering dendogram and the resulting stabil- 
ity cut. 

Since the subjects were instructed to sort the infor- 
mation by how they viewed the information’s similarity, 
the resulting clusters can be thought of as representing 
the pilots’ cognitive categories . These mental classifica- 
tions of information are used by individuals to reduce the 
amount of information with which they work (Woods 
1985). 7 Instead of dealing with each item of information 

7 As summarized in Clay (1993), page 40, Woods (1985) sug- 
gests that information should be organized based on high-level 
units and that task-meaningful units should be identified for orga- 
nization. In addition, information that must be processed together 
should be grouped together. 
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on the approach chart as an independent element, the 
pilots can categorize (organize by similarity clusters) the 
information, manage it through these categories, and then 
call on the individual item when it is needed. Pilots are 
then able to work comfortably with the approach chart 
without being overwhelmed by the amount of informa- 
tion it contains. 

Results of the cluster analysis showed that the pilots 
use 10 primary categories to organize and manage 
approach chart information. The 10 clusters that emerged 
(numbered from top to bottom in fig. 9) are shown in 
figure 10 and have been labeled with category interpreta- 
tions. Neither the order of the clusters nor the order of 
elements within clusters has any significance in this 
figure. 

The first cluster includes the ATIS and other fre- 
quencies that are used for verbal communications. Con- 
sequently, this category has been labeled Commun- 
ications. While many frequencies exist on the approach 
chart, only those in this category pertain to radios that are 
used for verbal communications. It is probably for this 
reason that pilots mentally separate the frequencies by 
the higher level tasks they support. As stated above, by 
categorizing the information, pilots should be able to 
determine that information is of a certain type (e.g., 
Communications) without individually addressing each 
element that comprises the category. Many decisions 
then can be made based solely on the category, thus less- 
ening the overall task workload. When the occasion 
arises, pilots are able to access specific information from 
the categories efficiently. 

The second cluster shown in the figure contains ele- 
ments pertaining to geographic positions and has been 
labeled Geography. Elements in this cluster pertain to the 
physical layout of the terrain surrounding the airport and 
runway. Members of this category were the latitude- 
longitude grid for the approach chart, the depiction of the 
airport VOR’s latitude-longitude, the magnetic variation, 
and the map scaling. Major geographic elements (e.g., 
contour lines) on the chart were not manipulated (i.e., 
used as stimuli or grayed on the approach chart) in this 
experiment; however, given this interpretation, they 
would be members of this cognitive category since they 
provide the general map of the surrounding terrain. 

The next cluster contains elements such as airport 
name and chart date, which appear to represent elements 
used in chart Validation. These items are used to assure 
the pilots that the chart is the correct one, that it is cur- 
rently valid, and that both pilots are on the same page. 
An exception to this interpretation might be airport ele- 
vation. However, its membership in this cluster may be 
because many of the crews were from American Air- 
lines, who by using the conversion factor given to them 


by dispatch to make one altimeter read zero at the air- 
port elevation, subsequently use airport elevation for 
validation. 

The fourth cluster has been labeled Obstructions 
since all elements in this cluster present either minimum 
safe altitudes or altitudes of obstructions. Members of 
this cluster were the highest reference point, minimum 
safe altitude (MSA) on approach procedure, MSA mini- 
mum altitude, MSA reference point, MSA sector radials, 
and obstructions. These element types differ from other 
altitude references on the chart (e.g., step-down altitude) 
by the task they support. Altitudes in the Obstructions 
category are lower limits (not to be met), whereas the 
other altitudes on the approach chart are target altitudes. 

The fifth and largest cluster contains the primary ele- 
ments that are used for planning and monitoring the 
approach descent and direction. While the other elements 
on the chart can be considered reference information, 
members of this cluster are the ones used for planning 
and monitoring the general task of navigation (both hori- 
zontal and profile). Large clusters are the most difficult 
to interpret. Since nine of the elements pertain to those 
found strictly in the profile section of the chart (vertical 
navigation) and seven are used for both profile and hori- 
zontal navigation, the tendency would be to give this cat- 
egory a name that captures only the profile theme. 
However, when lower level clustering was studied, it was 
obvious that the elements pertaining to the horizontal 
navigation were not arbitrarily being grouped with this 
information and should not be lost in the interpretation. 
A subsequent look at the elements making up this cluster 
led to the more general interpretation of Navigation. 
While the overall purpose of the approach chart is to pro- 
vide navigation information, this cluster was most repre- 
sentative of that function. It would have been more 
convenient had the vertical and horizontal elements 
remained separate (as they were at lower level cluster- 
ings) where the cut was made; however, the elements 
were grouped at this level because the pilots deemed 
them more similar to one another than to members of the 
other clusters. Perhaps this similarity exists because 
these elements are the primary ones used for planning the 
approach and monitoring its progress. 

The next cluster, labeled Missed Approach , contains 
elements that are of interest when pilots are executing 
a missed approach. Members of this cluster were 
holding-pattern course, location of missed approach 
point (MAP), MAP (nonprecision), MAP (precision), 
missed-approach instruction, and time from FAF to 
MAP. 

The seventh cluster contains Final Items that the 
pilots use or reference, such as the glideslope (GS) angle, 
rate of descent, threshold crossing height, and the 
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touchdown zone elevation (TDZE) of the target runway. 
Unlike items in the Navigation cluster that are used dur- 
ing the final segment, members of Final Items are used 
for reference instead of for planning and monitoring. 

While the information in cluster 7 deals with final 
items for the target runway, cluster 8 pertains to final 
items for other runways and has been labeled Other Run- 
ways. With the exception of runway length, these items 
probably emerged as a separate cluster for the pilots 
because they do not pertain to their approach. Including 
runway length in Other Runways may have been an arti- 
fact of the linkage method selected for this analysis. 

The next cluster contained only one element and was 
labeled accordingly, Visibility Requirement. This infor- 
mation stands out with pilots and subsequently did not 
cluster with other stimuli, perhaps because of its per- 
ceived importance, use, and frequent acquisition. 

The tenth and final cluster contained elements per- 
taining to navigation radios and has been labeled Naviga- 
tion Aids. This cluster contains more than just the 
frequencies for the navigation aids (in contrast to Com- 
munications). As the lower level clustering (fig. 9) and 
the acquisition data (described below) showed, pilots 
considered the navigation aid frequency, identifier, and 
Morse code as one “chunk” of information. This consid- 
eration was again apparent in the retrospective descrip- 
tions that were given by the pilots when they described 
why they acquired specific items. It is probably for this 
reason that these elements combined within one cluster. 

Multidimensional scaling analysis. In addition to 
the cluster analysis, the similarity data (i.e., the proximity 
matrix) were also analyzed using multidimensional scal- 
ing (MDS); MDS calculates a spatial representation 
among stimuli using the subjects’ measures of how the 
stimuli are related to one another. This spatial representa- 
tion presents the objects in an n-dimensional space, with 
items that are similar to one another lying close together, 
while dissimilar items lie farther apart in the space. The 
MDS analyses are used to determine whether the sub- 
jects’ underlying processes that are used as the basis for 
their information similarity ratings can be meaningfully 
represented spatially. For example, if pilots perceived 
information similarity by determining when an item is 
used and also by the section of the chart in which it 
appears, then a two-dimensional solution might result 
with items on the dimensions arranged according to these 
interpretations. For a more detailed description of MDS, 
refer to Kruskal and Wish (1978). 

The MAPLIST similarity data were scaled in one, 
two, and three dimensions using a nonmetric scaling pro- 
gram (Wilkinson, Hill, and Vang 1992). Adequacy of the 
solutions was based on both the reduction in STRESS 


values with the addition of each dimension (lower 
STRESS values indicate that the data fit better) and the 
interpretability of the obtained solution. For the current 
data, STRESS values were 0.283, 0.148, and 0.104 for 
the one-, two-, and three-dimensional solutions, respec- 
tively. The reduction in STRESS values appeared to 
level off at two dimensions, so this solution was retained 
for analysis. 

Because of the large number of stimuli, only a subset 
(of the stimuli) was used in this two-dimensional plot 
(for legibility). Since dimensional interpretation is en- 
hanced frequently by examining the stimuli that are lying 
on the extremes of the dimensions, the stimuli that were 
plotted constitute the low and high ends of each dimen- 
sion. In addition, several stimuli lying in the center of the 
solution were plotted. This spatial solution is shown in 
figure 1 1 . 

Examination of the plot revealed no obvious dimen- 
sional interpretation and might suggest that the data are 
not particularly well fit by a scaling solution. Given the 
well-defined clusters, this is not particularly surprising. 
The clustering results show that pilots have well- 
established categories into which they place flight deck 
information. In such cases spatial representation may be 
inappropriate for representing the data, thereby leading 
to a poor spatial fit. As Shepard (1980, p. 397) has noted, 
“While both clustering and MDS analyses are useful for 
exploring some sets of data, different models may be 
more appropriate for different sets of stimuli or types of 
data.” In the present case, the proximity data appear best 
suited to a nonspatial clustering representation, as pre- 
sented above. While no interpretation was rendered in 
this experiment, the results and discussion were included 
for completeness and to afford the reader an opportunity 
to interpret the results. 

Importance of Approach Chart Information 

Elements 

As described in the “Methods” section, subjects 
were asked to rank the 75 information elements by per- 
ceived importance. The pilots were first instructed to 
rank the elements without a given context (context free). 
Immediately after the context-free ranking, the pilots 
were asked to rank the elements again, based on the 
importance of the information pilots use when they are 
flying approaches in adverse weather. Adverse weather 
was described as both visibility and cloud bottom at min- 
imums. Average rankings were calculated for the two 
test conditions, and multidimensional preference analy- 
ses were done on these rankings. Descriptions of these 
analyses and results are presented below. 

Average rankings. The average ranking of elements 
for both the context-free and bad weather context is 
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given in table 3. The subjects assigned low numbers to 
elements that they perceived to be the most important. 
This numbering indicates that elements such as ILS 
localizer magnetic course and visibility requirement are 
considered to be very important whereas magnetic varia- 
tion and the TDZE of the parallel runway are of lesser 
importance. An observation in addition to the relative 
order of elements within a context is the relative consis- 
tency of element rankings between contexts (i.e., context 
free and adverse weather). This ranking was demon- 
strated by a correlation coefficient of 0.983, which indi- 
cated that there was a very high level of agreement on the 
importance of each information element across these two 
different contexts. 

Since the relative order of the stimuli was very close, 
it is possible that the importance of the information on 
approach charts is insensitive to context or is somewhat 
absolute. The results could be explained also by the con- 
text in which some pilots defaulted when they were not 
given a context (i.e., defaulting to adverse weather condi- 
tions during the context-free ranking). While neither 
explanation is definitive, it can be said that the context 
that was used in this study did not affect the pilots’ per- 
ception of the importance of this information (as 
reflected in the data). 

In studying the average rankings, it was observed 
that many of the element types that were identified as 
highly important come from the similarity categories of 
Visibility Requirement (“Are we legal to fly this 
approach?”), and chart Validation (“Do we have the cor- 
rect and same approach?”). Since these elements repre- 
sent items checked at the very beginning of the approach, 
there may have been a temporal factor used when impor- 
tance was assigned. 

That Geography and Other Runways information 
occupied the “least important” positions is also notewor- 
thy. From this observation it is fairly safe to say that 
some elements identified as least important were named 
so because they did not pertain to this flight (i.e., Other 
Runways). With the exception of the element runway 
length (which was also an exception to the similarity cat- 
egory), each element in the Other Runways category was 
deemed unimportant. These elements were the TDZE of 
the parallel runway, the glideslope position on the paral- 
lel runway, and other runway numbers. 

Geography was another similarity category in which 
elements were deemed of lesser importance by the pilots. 
Each element from the Geography category, even though 
it was pertinent to the approach, was deemed least impor- 
tant by the pilots when they ranked the elements. As a 
reminder, these elements were latitude-longitude grid, 
latitude-longitude of airport VOR, magnetic variation, 
and scaling. 


Multidimensional preference analysis. The ele- 
ment rankings for all subjects were also analyzed by 
using multidimensional preference (MDPREF) (Carroll 
1972); MDPREF is a multidimensional-preference anal- 
ysis program designed to accommodate ranking data. 
The MDPREF is a technique that takes ranking stimulus 
data and provides a joint representation of the data and 
subject preferences. The analysis presents the stimuli 
spatially, analogous to MDS (discussed earlier). The 
MDPREF analysis was conducted using the personal 
computer (multidimensional) preference (PCPREF) al- 
gorithm, a personal computer version of an MDPREF 
analysis. (For further details on the model, see Carroll 
1972.) An MDPREF analysis was performed on both the 
context-free and adverse weather conditions. 

A dimension in an MDPREF solution corresponds to 
a factor that subjects use to process the information (i.e., 
determine its importance). Subjects often use multiple 
factors when making their ratings. For example, pilots 
may determine the importance of information according 
to when it is needed in the flight and also by logical 
dependencies. In such a case a two-dimensional solution 
would result. 

In determining the number of significant factors (i.e., 
the dimensionality), an examination is made of the pro- 
portion of variance that is accounted for by each factor. 
The larger the variance, the more a given factor is able to 
reproduce the original data matrix; hence, the factor can 
be thought of as representing whatever common structure 
exists, to a greater degree, in the underlying data. Partic- 
ular attention is paid to relative decreases in variance that 
could be accounted for as more dimensions are added 
and that would ease interpretation of the solution. 

Proportion of variance values showed that a one- 
dimensional solution of the context-free data accounted 
for 0.092 of the variance. Adding a second dimension 
accounted for an additional 0.081 of the variance 
(total = 0.173). The total proportion of variance when 
adding a third dimension was 0.249, and it was 0.321 
after adding a fourth dimension. Cumulative proportion 
of variance declined at a slightly slower rate thereafter 
(five dimensions added 0.059 and six dimensions added 
another 0.056). 

These values are very low and indicate that the data 
did not conform well to a spatial solution. This result can 
be explained by the subjects use of many different factors 
when they were making their ratings. Since the primary 
goal of an MDPREF analysis is to reduce the number of 
factors to a smaller set that accounts for a fairly large 
proportion of the variance, these results indicate a poor 
fit. Therefore, for this input, a “good” dimensional solu- 
tion was not discernible. 
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Table 3. Average Rankings 


Element, context free 

Average 

ranking 

(a) 

Element, context free 

Average 

ranking 

(a) 

Element, adverse 
weather context 

Average 

ranking 

(a) 

Element, adverse 
weather context 

Average 

ranking 

(a) 

ILS localizer 
magnetic course 
Visibility 
requirement 
Decision altitude 
(AGL) 

Effective date 
Procedure name 
ILS localizer 
frequency 
ILS identifier 
Airport elevation 
Airport name 
Chart date 
Runway length 
MAP (precision) 
Decision altitude 
(MSL) 

Inbound course 
ILS Morse code 
GS intercept altitude 
(MSL) 

FAF intercept altitude 
(MSL) 

Location of MAP 
FAF intercept altitude 
(AGL) 

GS intercept altitude 
(AGL) 

MAP (nonprecision) 
Obstruction 
MSA minimum 
altitude 

VOR frequency 
Time from FAF to 
MAP 

Missed approach 
instruction 
DME distance 
Fix name at FAF 
Outbound course 
Step-down altitude 
(MSL) 

Min safe altitude on 
AP 

Procedure turn 
altitude 

VOR identifier 
Fix name at IAF 
Visual descent point 
MSA reference point 
MSA sector radials 
Primary facility 
frequency 

16.2 

17.2 
20.1 

20.8 

21.2 

21.5 

21.7 

21.7 

22.0 

24.0 

24.1 

25.0 

25.6 

25.7 

26.3 

27.0 

27.1 

28.4 

28.5 

30.2 

30.3 

30.4 

30.6 

30.7 

31.4 

32.1 

34.1 
34.1 
34.1 

34.7 

35.3 

35.4 

36.3 

36.8 

36.8 

36.9 

37.4 
38.0 

Distance (non-DME) 
VOR Morse code 
NDB frequency 
Procedure turn 
distance limit 
Step-down altitude 
(AGL) 

Approach frequency 
Fix name 
Holding pattern 
course 

Chart index number 
NDB Morse code 
Rate of descent 
ATIS arrival 
frequency 
Highest reference 
point 

Lead radial 
NDB identifier 
VOR name 
Feeder route radial 
GS angle 

Threshold crossing 
height 

Primary facility 
Morse code 
TDZE target runway 
Tower frequency 
Geographic name 
Notes 

Ground frequency 
GS position on target 
runway 
Changes 
Other runway 
numbers 

Airport departure 
frequency 
ATIS clearance 
frequency 
Lak/long of airport 
VOR 

ATIS departure 
frequency 
Scaling 

GS position on 
parallel runway 
TDZE parallel 
runway 
Lat./long grid 
Magnetic variation 

38.1 
38.3 
38.6 

38.6 

38.8 

38.9 

39.2 

40.0 

41.1 

41.2 

41.9 
42.1 

43.0 

43.0 

43.1 
43.1 

43.3 

43.8 

43.9 

44.0 

44.3 

44.4 

44.9 

45.5 

49.0 

51.1 

52.3 

54.3 

55.8 

56.1 

59.4 

59.4 

61.7 

63.5 

64.1 

64.6 

65.7 

Visibility 
requirement 
ILS localizer 
magnetic course 
Procedure name 
Decision altitude 
(AGL) 

Runway length 
Airport elevation 
Airport name 
ILS localizer 
frequency 
Effective date 
ILS identifier 
Decision altitude 
(MSL) 

Chart date 
MAP (precision) 

ILS Morse code 
MSA minimum 
altitude 

Inbound course 
GS intercept altitude 
(MSL) 

Missed approach 
instruction 

FAF intercept altitude 
(MSL) 

FAF intercept altitude 
(AGL) 

GS intercept altitude 
(AGL) 

Obstruction 
Min safe altitude on 
AP 

Location of MAP 
VOR frequency 
Time from FAF to 
MAP 

MSA reference point 
MAP (nonprecision) 
MSA sector radials 
Step-down altitude 
(MSL) 

Fix name at FAF 
Outbound course 
DME distance 
Fix named at IAF 
VOR identifier 
Highest reference 
point 

Procedure turn 
distance limit 
Holding pattern 
course 

10.2 

14.2 

17.0 

17.0 

18.4 

19.0 

19.3 
19.9 

20.5 

21.4 

22.7 

22.8 

23.3 

23.6 
24.8 

25.5 

25.7 

26.4 

26.6 

28.0 

28.8 

29.1 

30.2 

30.5 

30.7 

32.0 

32.1 

32.3 

32.4 
34.0 

34.0 

34.6 
34.6 

36.0 

37.0 

37.4 

37.8 

38.0 

Procedure turn 
altitude 

Primary facility 
frequency 
VOR Morse code 
Visual descent point 
Step-down altitude 
(AGL) 

Approach frequency 
VOR name 
NDB frequency 
ATIS arrival 
frequency 

Distance (non-DME) 
Rate of descent 
Fix name 
GS angle 

Chart index number 
Lead radial 
Threshold crossing 
height 

NDB Morse code 
NDB identifier 
Feeder route radial 
TDZE target runway 
Geographic name 
Primary facility 
Morse code 
Tower frequency 
Notes 

GS position on target 
runway 
Changes 

Ground frequency 
Other runway 
numbers 

Airport departure 
frequency 
GS position on 
parallel runway 
ATIS departure 
frequency 
Lat./long of airport 
VOR 

ATIS clearance 
frequency 
Scaling 
Lat./long grid 
TDZE parallel 
runway 

Magnetic variation 

38.4 

38.9 

39.3 

39.8 

40.2 

40.4 

40.9 

40.9 

41.0 

41.5 

42.3 

42.4 

42.9 

43.1 

43.1 

43.6 

44.1 

44.6 

45.4 

45.9 

46.7 

47.1 

47.9 

47.9 

48.9 

52.1 

55.4 

58.8 

61.8 

62.4 
63.7 

63.7 

64.4 

64.7 

64.8 

65.5 

67.8 


a Lower values indicate higher perceived priority. 
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The same was true in the adverse weather condition. 
As with the context-free condition, proportion -of- 
variance values indicated that discriminating among dif- 
ferent dimensional solutions was not possible. The one- 
dimensional solution accounted for 0.095 of the variance 
and adding a second dimension accounted for an addi- 
tional 0.077 of the variance (total = 0. 1 72). The total pro- 
portion of variance when adding a third dimension was 
0.246, and it was 0.315 after adding a fourth dimension. 
Cumulative proportion of variance declined at a slightly 
slower rate thereafter (five dimensions added 0.059, and 
six dimensions added another 0.056). 

Although the above results indicated that these data 
might not be represented best spatially, an attempt was 
still made (by examining a few of the spatial plots) to 
interpret the factors the pilots used when they were mak- 
ing their importance rankings. Since solutions for the 
context-free and adverse weather rankings were nearly 
identical, the discussion below will focus only on the 
context-free condition. To aid in this investigation and 
for discussion concerning what the pilots may have been 
using to differentiate between the stimuli during this 
task, the three-dimensional solution was used and is plot- 
ted (fig. 12) using those stimuli that define the dimen- 
sional extremes. When looking at the three-dimensional 
plot, it appeared as though the pilots made their rankings 
based on their temporal position in the approach. That is, 
their rankings differed depending on whether or not the 
pilots envisioned themselves to be in the preapproach, 
approach transition, intermediate, or final segment. 

For the first dimension the ends were defined by Vis- 
ibility Requirement and decision altitude (MSL) at one 
end and location of missed approach point (MAP) and 
glideslope (GS) angle at the other. Elements appear to be 
ordered along this dimension based on their importance 
to the preapproach phase of the approach. In the pre- 
approach phase, pilots are more concerned with whether 
they are legal to fly the approach (i.e., comparing visibil- 
ity and cloud bottom values given by ATIS to the Visi- 
bility Requirement and decision altitude listed for the 
approach) than they are for elements of interest during 
the final segment, such as the glideslope angle and the 
location of the MAP. 

The second dimension is defined by elements such 
as non-DME distances and the MSA reference point at 
one end and TDZE’s of both the target and parallel run- 
ways at the other end. The most important elements on 
this dimension seem to be those of most interest during 
the transition to intermediate segments of the flight. Dur- 
ing these segments pilots are planning, using the mileage 
presented (nondistance measuring equipment (DME) dis- 
tances) and the minimum altitudes, as well as constantly 
monitoring both of these items when they are flying the 
feeder radials. 


The elements that define both extremes of the third 
dimension seem to be most important, with the distinc- 
tion being the temporal position within the final segment 
(i.e., beginning or end of final approach). Glideslope 
intercept altitude, rate of descent, and runway length are 
most important to the pilots when they are beginning the 
final segment of the approach. During this segment, the 
PNF checks the altitude when intercepting the glideslope 
against the altitude depicted on the chart. Rate of descent 
is an element of constant monitoring during this phase, 
especially when the pilots are flying VOR (i.e., no 
glideslope signal to provide this information). In the 
event that either the rate of descent is too slow or the 
point of intercept of the glideslope is too high, crews 
must ascertain that there is still enough runway (i.e., run- 
way length) to land or adjust their rate of descent. Items 
on the other end of this dimension, such as decision alti- 
tude (MSL) and Visibility Requirement, are most impor- 
tant toward the end of the final segment when a final 
decision must be made that concerns continuing the 
approach or initiating missed approach procedures. 

Pilots’ Acquisition of Approach Information 

All information items that could be selected on the 
approach charts by the pilots during the simulation por- 
tion of the experiment were identified by their corre- 
sponding information element types (table 1) for post 
analysis. Segments of the approach in which they were 
selected were identified for further delineation. 

The preapproach phase was not a time-controlled 
segment. Each crew was told to take the time it needed 
to get familiar with the approach and for the PF to 
brief the PNF. Average time for the preapproach phase 
was 315 seconds with crew averages ranging from 
211 seconds to 504 seconds. Transition, intermediate, 
and final were time-controlled segments of the approach 
(i.e., duration within segments was controlled by ATC). 
Average time in the transition phase was 626 seconds. 
Average time in the intermediate phase was 79 seconds, 
and average time in the final segment was 139 seconds. 
For more information on the time profiles, please refer to 
the tables in appendix B. 

This section begins with a discussion of the possible 
limiting factors and presents the descriptive results that 
concern the characteristics of the pilots’ retrieval of 
information during the simulated approaches. Inferential 
analyses also were done by using the derived clusters 
(i.e., similarity categories) as dependent measures when 
the acquisition test matrix that is described in the 
“Method” section was being analyzed. These results will 
be described in the “Effects of Independent Variables on 
Pilot Acquisition” section following the descriptive 
analyses. 
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Related Debriefing Questionnaire Responses 

One question in the debriefing questionnaire per- 
tained to the effect of the pilots’ doing the psychometric 
scaling tasks before their simulation session. The intent 
of the question was to determine whether the pilots 
changed the way they did things in the simulator because 
of their perception of the experiment’s objectives. The 
pilots were asked whether or not their experience in the 
card tasks changed their information acquisition strategy 
in the simulation task. Five said that they did change 
their strategies based on the psychometric scaling tasks 
and 21 said they did not. Assessment of the five pilots’ 
comments indicated that there was no real impact on the 
study’s objectives. Pilot comments have been included 
below for completeness. 

“I realized that there was some important infor- 
mation that I was overlooking; however, the 
general flow of information in the acquisition 
strategy did not change.” 

“It made me try to be more time-information 
efficient and to prioritize to a greater degree.” 

“Possibly there was an impact. The card task 
heightened awareness of some items that are 
normally provided or taken for granted (e.g., air- 
port name).” 

“I usually forget all about MSA’s at Chicago- 
O’Hare (ORD) and Dallas-Fort Worth (DFW). 
However, seeing the cards brought it to mind, 
and I was more aware of it for the sim.” 

“I focused attention on what I was doing to 
verify my card task decision.” 

In addition, the pilots were asked many questions 
pertaining to the acquisition task and the simulation ses- 
sion in general. Answers to these questions and the 
pilots’ comments were useful in the analyses of these 
data and provided many insights. A compilation of these 
questions and comments is included in the discussion 
provided in appendix C. 

Descriptive Analyses 

As described above, the subjects’ acquisitions of 
approach chart information were time stamped and 
logged with corresponding flight segment information. 
These acquisition counts of the information items were 
mapped to the element types and graphed against the 
flight segments. Each row of a graph corresponds to an 
information element type (ordered alphabetically) and 
each column is a flight segment. A summary graph of the 
acquisitions from all the runs was made and is shown in 
figure 13. Graphs were also generated for acquisitions by 
the PF (fig. 14) and PNF (fig. 15), as well as for each 


level of the independent variables: weather and preci- 
sion. Good weather acquisitions are shown in figure 16; 
adverse weather acquisitions are presented in figure 17. 
For precision, ILS approach acquisitions are shown in 
figure 18, and VOR approach acquisitions are plotted in 
figure 19. 

In figure 13 it is apparent that most of the acquisi- 
tions are made during the preapproach phase of the 
flight. Sixty percent of all information acquired was 
selected during this phase in preparation for the approach 
(i.e., before initial descent). Thirty-one percent of the 
selections were made during the approach transition 
phase. Five percent of the selections were made during 
the intermediate phase. Four percent of the selections 
were made during the final segment. 8 These acquisition 
patterns appear inversely proportional to the number of 
flying tasks to be performed. As the number of flying 
tasks increased, the number of approach chart acquisi- 
tions decreased. These data may support the theory that 
the approach chart is primarily used for planning. 

Table 4 repeats the data in the last column of fig- 
ure 1 3 and orders the elements according to frequency of 
acquisitions. Elements that were omitted from the table 
were never acquired. Apparent from this presentation of 
the data is that the first 9 of the information types that 
were listed accounted for over 50 percent of the overall 
acquisitions but that those 9 types represented less than 
26 percent of the total number of items that could be 
selected on the charts. Feeder route radial (1.81 percent 
of total items) was selected 8.28 percent of the time; min- 
imum safe altitude (MSA) on approach procedure 
(2.07 percent of total items) was selected 7.81 percent of 
the time; DME distance (3.89 percent of total items) was 
selected 6.33 percent of the time; VOR identifier 
(3.63 percent of total items) was selected 6.1 1 percent of 
the time; distance (non-DME) (4.40 percent of total 
items) was acquired 5.88 percent of the time; fix name at 
FAF (2.07 percent of total items) was acquired 4.40 per- 
cent of the time; VOR frequency (1.42 percent of total 
items) was acquired 4.35 percent of the time; ILS identi- 
fier (3.63 percent of total items) was selected 4.33 per- 
cent of the time; and fix name (2.85 percent of items) 
was acquired 3.95 percent of the time. 

As stated above, most of the acquisitions were made 
during the preapproach segment of the simulation run (on 
average, 61 percent of all acquisitions). Tables 5 through 
8 show the acquisition totals for preapproach and each 
approach segment, except for the missed approach seg- 
ment, ranked by number of acquisitions. Together these 

8 Missed Approach was not an intended segment for this study; 
however, some crews initiated missed approaches which accounted 
for a small fraction of the total number of items selected. 
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Table 4. Total Acquisitions 


Information element 

Acquisition 

Information element 

Acquisition 

Information element 

Acquisition 

Feeder route radial 

956 

VOR Morse code 

149 

Primary facility Morse code 

38 

MSA on approach proc. 

902 

ILS localizer mag. course 

148 

Lat-long grid 

30 

DME distance 

731 

FAF intercept alt. (AGL) 

139 

TDZE parallel runway 

30 

VOR identifier 

706 

Geographic name 

132 

MSA reference point 

29 

Distance (non-DME) 

679 

Airport name 

130 

Procedure turn dist. limit 

29 

Fix name at FAF 

508 

Obstruction 

128 

GS intercept alt. (AGL) 

28 

VOR freq. 

503 

Chart index number 

116 

Approach freq. 

27 

ILS identifier 

500 

Chart date 

110 

MSA sector radials 

25 

Fix name 

456 

Holding pattern course 

105 

Ground freq. 

19 

Procedure name 

324 

Outbound course 

98 

MM crossing alt. (MSL) a 

15 

Decision alt. (MSL) 

306 

Location of MAP 

97 

Rate of descent 

15 

Inbound course 

299 

GS intercept alt. (MSL) 

94 

MM crossing alt. (AGL) a 

10 

FAF intercept alt. (MSL) 

279 

ILS Morse code 

80 

Notes 

— — - — i 

10 

Missed approach instruction 

239 

Time from FAF to MAP 

80 

GS angle 

8 

Visibility requirement 

239 

Runway length 

65 

Lat-long of airport VOR 

8 

Fix name at IAF 

233 

Step-down alt. (AGL) 

65 

ATIS arrival freq. 

6 

VOR name 

199 

Other runway numbers 

58 

GS position on target rwy 

6 

Airport elevation 

194 

TDZE target runway 

50 

Airport departure freq. 

2 

MSA minimum alt. 

190 

MAP (nonprecision) 

47 

ATIS departure freq. 

2 

Decision alt. (AGL) 

189 

Highest reference point 

45 

ATIS clearance freq. 

1 

Step-down alt. (MSL) 

178 

Lead radial 

45 

Magnetic variation 

1 

ILS localizer freq. 

175 

Tower freq. 

45 

Threshold cross height 

1 

Primary facility freq. 

161 

Effective date 

38 




a Element was not part of the psychometric scaling stimuli. 


Table 5. Preapproach Acquisitions 


Information element 

Acquisition 

Information element 

Acquisition 

Information element 

Acquisition 

Feeder route radial 

464 

Geographic name 

128 

ILS Morse code 

37 

VOR identifier 

437 

Airport name 

124 

TDZE target runway 

37 

MSA on approach proc. 

398 

MSA minimum alt. 

118 

MAP (nonprecision) 

36 

DME distance 

350 

Chart index number 

115 

Lead radial 

32 

VOR freq. 

347 

Chart date 

110 

Primary facility Morse code 

29 

ILS identifier 

291 

Step-down alt. (MSL) 

107 

Highest reference point 

25 

Procedure name 

290 

VOR Morse code 

106 

MSA reference point 

23 

Distance (non-DME) 

282 

Holding pattern course 

95 

TDZE parallel runway 

21 

Fix name at FAF 

280 

FAF intercept alt. (AGL) 

91 

GS intercept alt. (AGL) 

19 

Visibility requirement 

220 

ILS localizer freq. 

75 

MM crossing alt. (MSL) a 

13 

Fix name 

216 

ILS localizer mag. course 

69 

MSA sector radials 

13 

Fix name at IAF 

181 

Runway length 

62 

Procedure turn dist. limit 

13 

Decision alt. (MSL) 

179 

Location of MAP 

60 

Lat-long grid 

12 

Missed approach instruction 

157 

Obstruction 

59 

Rate of descent 

9 

Decision alt. (AGL) 

151 

Time from FAF to MAP 

58 

MM crossing alt. (AGL) a 

8 

FAF intercept alt. (MSL) 

150 

GS intercept alt. (MSL) 

53 

Notes 

8 

Inbound course 

148 

Outbound course 

48 

Lat-long of airport VOR 

6 

Primary facility freq. 

147 

Other runway numbers 

42 

GS angle 

5 

VOR name 

143 

Step-down alt. (AGL) 

38 

GS pos. on target runway 

4 

Airport elevation 

136 

Effective date 

37 




a Element was not part of the psychometric scaling stimuli. 
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Table 6. Transition Phase Acquisitions 


Information element 

Acquisition 

Information element 

Acquisition 

Information element 

Acquisition 

Feeder route radial 

487 

Outbound course 

32 

Visibility requirement 

10 

MSA on approach proc. 

485 

Procedure name 

26 

Primary facility Morse code 

8 

Distance (non-DME) 

381 

Step-down alt. (AGL) 

24 

MAP (nonprecision) 

6 

DME distance 

248 

Tower freq. 

22 

MSA reference point 

6 

Fix name 

217 

Approach freq. 

21 

Rate of descent 

6 

VOR identifier 

179 

GS intercept alt. (MSL) 

21 

Airport name 

5 

ILS identifier 

148 

Highest reference point 

20 

TDZE target runway 

4 

VOR freq. 

139 

FAF intercept alt. (AGL) 

19 

ATIS arrival freq. 

3 

Fix name at FAF 

128 

Lat-Iong grid 

18 

GS angle 

3 

Inbound course 

117 

Time from FAF to MAP 

18 

GS intercept alt. (AGL) 

3 

ILS localizer freq. 

89 

Ground freq. 

17 

Geographic name 

3 

ILS localizer mag. course 

74 

Missed approach instruction 

16 

Runway length 

3 

MSA minimum alt. 

72 

Proc. turn distance limit 

16 

TDZE parallel runway 

3 

Obstruction 

67 

Other runway numbers 

15 

Airport departure freq. 

2 

Step-down alt. (MSL) 

65 

Decision alt. (MSL) 

14 

GS pos. on target runway 

2 

VOR name 

52 

Location of MAP 

14 

MM crossing alt. (AGL) a 

2 

Fix name at IAF 

47 

Primary facility freq. 

13 

MM crossing alt. (MSL) a 

2 

FAF intercept alt. (MSL) 

45 

Lead radial 

12 

Notes 

2 

ILS Morse code 

40 

MSA sector radials 

12 

Chart index number 

1 

VOR Morse code 

40 

Decision alt. (AGL) 

11 

Effective date 

1 

Airport elevation 

38 

Holding pattern course 

10 

Lat-long of airport VOR 

1 


a Element was not part of the psychometric scaling stimuli. 


Table 7. Intermediate Phase Acquisitions 


Information element 

Acquisition 

Information element 

Acquisition 

Information element 

Acquisition 

DME distance 


Missed approach instruction 

10 

Step-down alt. (AGL) 

3 

Fix name at FAF 

80 


9 

VOR name 

3 

FAF intercept alt. (MSL) 

64 

ILS localizer freq. 

9 

Fix name at IAF 

1 

VOR identifier 

52 

VOR freq. 

9 

ILS Morse code 

1 

ILS identifier 

40 

Step-down alt. (MSL) 

6 

Lat-long of airport VOR 

1 

Inbound course 

31 

Decision alt. (AGL) 

5 

Location of MAP 

1 

Decision alt. (MSL) 

26 

Airport elevation 

MKIf 

Magnetic variation 

1 

Fix name 

22 

Approach freq. 


Primary facility freq. 

1 

FAF intercept alt. (AGL) 

20 

GS intercept alt. (AGL) 


Procedure name 

1 

GS intercept alt. (MSL) 

17 

ILS localizer mag. course 


TDZE parallel runway 

1 

Tower freq. 

17 

Visibility requirement 


TDZE target runway 

1 

MSA on approach proc. 

13 

Feeder route radial 


VOR Morse code 

1 

Outbound course 

13 






indicate that different information elements are accessed 
most during different segments of flight. Elements that 
were omitted from the tables were those that were not 
acquired during the respective phase. 

In the preapproach phase, 52 percent of the acquisi- 
tions were of items commonly used for validating and 
planning the approach. These 1 1 element types repre- 


sented only 35 percent of the items that could have 
been selected. Since most of the acquisitions were done 
in the preapproach segment, these acquisitions domi- 
nated the summary data presented above. In addition to 
the 9 dominant elements listed in the above summary 
table, the preapproach segment had procedure name 
and visibility requirement in the top 50 percent of 
acquisitions. 
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Table 8. Final Phase Acquisitions 


Information element 

Acquisition 

Information element 

Acquisition 

Information element 

Acquisition 

Decision alt. (MSL) 

87 

DME distance 

7 

GS intercept alt. (AGL) 

2 

Missed approach instruction 

52 

Procedure name 

7 

ILS localizer freq. 

2 

VOR identifier 

38 

MSA on approach proc. 

6 

ILS Morse code 

2 

Distance (non-DME) 

23 

MAP (nonprecision) 

5 

Obstruction 

2 

Decision alt. (AGL) 

22 

Outbound course 

5 

VOR Morse code 

2 

Location of MAP 

22 

TDZE parallel runway 

■■ 

Airport name 

1 

ILS identifier 

21 

Tower freq. 

5 

ATIS departure freq. 

1 

FAF intercept alt. (MSL) 

20 

Visibility requirement 

5 

Fix name 

1 

Fix name at FAF 

20 

Fix name at IAF 

4 

ILS localizer mag. course 

1 

Airport elevation 

16 

Time from FAF to MAP 

4 

Lead radial 

1 

FAF intercept ait. (AGL) 

9 

GS intercept alt. (MSL) 

3 

Other runway numbers 

1 

TDZE target runway 

8 

Inbound course 

3 

Primary facility freq. 

1 

VOR freq. 

8 

Feeder route radial 

2 

VOR name 

1 


In the transition segment of the approach, 5 element 
types were acquired over 50 percent of the time, but they 
represented only 15 percent of the total number of items 
that could be selected. Feeder route radial (1.81 percent 
of total items) was selected 13.51 percent of the time; 
minimum safe altitude on approach procedure (2.07 per- 
cent of total items) was selected 13.45 percent of the 
time; distance (non-DME) (4.40 percent of total items) 
was selected 10.57 percent of the time; DME distance 
(3.89 percent of total items) was selected 6.88 percent of 
the time; and fix name (2.85 percent of total items) was 
selected 6.02 percent of the time. There were many com- 
mon items with those dominating the preapproach-phase 
acquisitions that may suggest that the transition phase is 
used primarily as a continuation of the planning process. 

Acquisitions in the intermediate phase were domi- 
nated by four information element types. Over 50 percent 
of the intermediate phase acquisitions were of elements 
that represented only 10.1 1 percent of all selection items. 
The DME distance (3.89 percent of total items) was 
acquired 18.58 percent of the time; fix name at FAF 
(2.07 percent of total items) was acquired 13.51 percent 
of the time; FAF intercept altitude (MSL) (0.52 percent 
of total items) was selected 10.81 percent of the time; 
and VOR identifier (3.63 percent of total items) was 
selected 8.78 percent of the time. The tasks that these 
items support might suggest a transition during this seg- 
ment to monitoring progress of the approach (versus 
planning). 

Acquisition counts were dominated during the final 
segment by missed approach and distance information. 
While on final, 5 of the information element types were 
selected more than 52 percent of the time, but they only 
represented 15 percent of the total number of selection 
items. Decision altitude (MSL) (3.37 percent of total 


items) was selected 20.57 percent of the time; missed 
approach instruction (0.52 percent of total items) was 
selected 12.29 percent of the time; VOR identifier 
(3.63 percent of total items) was selected 8.98 percent of 
the time; distance (non-DME) (4.40 percent of total 
items) was selected 5.44 percent of the time; and deci- 
sion altitude (AGL) (3.37 percent of total items) was 
selected 5.20 percent of the time. Not part of the 52 per- 
cent of acquisitions during this phase, but of an equal 
number of acquisitions as decision altitude (AGL), was 
location of MAP (1.04 percent of total items), which was 
selected 5.20 percent of the time. These items again 
appear to support the pilots’ monitoring task, except for 
the Missed Approach information. 

Figures 13 and 14 chart the acquisitions made by all 
PF’s and by all PNF’s. The PNF’s made more acquisi- 
tions in each segment of flight than did the PF. The dif- 
ference between the two was more pronounced during 
the flying segments. Overall, the PNF accounted for 
65 percent of all the information items that were ac- 
quired. During the preapproach segment the PNF’s 
acquisitions totaled 56 percent, and during the flying seg- 
ments they accounted for 78 percent of the total number 
of information item acquisitions. 

The PF was expected to have fewer acquisitions 
overall than the PNF because the PF’s primary attention 
would be the flying tasks, while the PNF would handle 
the retrieval of approach chart information to support 
both the PF’s and the PNF’s planning and monitoring 
tasks. Two facts support this hypothesis: The PF got 
most of his information during the preapproach phase 
and the PNF retrieved more information during the flying 
phase. Responses to questions during the debriefing con- 
firmed that this acquisition pattern is common in actual 
airline flying. 
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Figures 18 and 19 show graphs of acquisition counts 
that were logged during the precision (ILS) and nonprec- 
ision (VOR) scenarios. In each flight segment, crews 
made more acquisitions in the nonprecision scenarios 
than they made in the precision scenarios (when counts 
pertaining solely to precision approaches are excepted). 
There were notably more acquisitions in the VOR totals 
for information elements: fix name at FAF, inbound 
course, minimum safe altitude on approach, distance 
(non-DME), and time from FAF to MAP. The ILS sce- 
narios produced considerably more acquisitions of the 
element fix name than did the VOR scenarios. Since 
much of the descent guidance is calculated automatically 
(e.g., glideslope) during the precision approaches and 
manually during the nonprecision approaches, it was 
expected that the number of acquisitions would be 
greater for the latter. The specific items that were 
accessed most during the nonprecision approaches sup- 
ported that expectation. 

The hypothesis that more acquisitions would be 
made during approaches in adverse weather than in good 
weather was valid. In adverse weather, the pilots are 
more concerned with such things as whether they are 
legal to fly the approach and distance to decision points. 
Figures 16 and 17 show the acquisition frequency graphs 
of the good and adverse weather scenarios. In all flight 
segments the subjects made more acquisitions in the 
adverse weather scenarios. The most notable differences 
were in the distance (non-DME) and the Visibility 
Requirement counts. 

Effects of Independent Variables on Pilot 

Acquisition 

In addition to the descriptive analyses discussed 
above, inferential analyses that used the derived similar- 
ity categories as dependent variables were done for the 
experiment design described in table 2 of the “Method” 
section. Acquisition of the information that comprises 
each category was sorted into their respective treatment 
combination. Since each subject participated in all com- 
binations of the treatments of precision (ILS and VOR), 
weather (good and adverse), and flight phase (pre- 
approach and approach), there were eight treatment 
combinations. 

A computer program was written to sort the number 
of information elements from the given category for each 
of 8 treatment combinations and for each of the 10 cate- 
gories. This sorting procedure allowed analyses of vari- 
ance (ANOVA) to be done for each of the 10 information 
categories, thereby rendering inferential statistical tests 
to observe the effects of the treatment conditions on the 
pilots’ acquisition behavior. Some of the statistically sig- 
nificant results from these analyses will be presented in 


terms of the respective main effects (crew member, pre- 
cision, weather, and flight phase) in conjunction with sta- 
tistically significant interactions where they are 
appropriate. In addition to this selective reporting of the 
ANOVA results, the complete ANOVA summary tables 
are presented in appendix D. Means and standard devia- 
tions for the respective ANOVA’s are shown in 
appendix E. 

Crew member. As expected from the descriptive 
results presented earlier, the PNF acquired more infor- 
mation from the approach chart than did the PF for all 
10 information categories. This result was statistically 
significant at the 5-percent level for the Navigation, Nav- 
igation Aids, Obstructions, and Missed Approach catego- 
ries. As discussed above, the PF’s primary interaction 
with the approach chart was during the preapproach 
phase, and at this point the PF and PNF both acquired 
about the same amount of information from the charts. 
However, when the flight ensues, the PF depends on the 
PNF to obtain the information from the approach chart. 
Several of the other information categories contained 
interactions that involve the crew member with another 
factor. These will be discussed in further detail in the 
appropriate section below. 

Precision . Statistically significant main effects for 
precision (VOR versus ILS) were observed for several of 
the information categories. More Navigation category 
information was obtained in the VOR condition than in 
the ILS condition, F(l, 22) = 14.2; probability (p) < 0.01 . 
This was probably due to the necessity of the pilots to 
manually calculate and monitor their descent. The same 
pattern was observed also for Obstructions information, 
F(l, 22) = 12.15, p< 0.01, Missed Approach informa- 
tion, F( 1, 22) = 29.96, p< 0.01, and for Visibility Re- 
quirement, F(l, 22) = 9.52, p<0.01. This pattern is 
attributable to the task differentiation between precision 
and nonprecision approaches. With VOR approaches, the 
pilots manually calculate and monitor the descent. 
Furthermore, their concern for Obstructions, Missed 
Approach information, and Visibility Requirements 
heightens when they are not given the automated guid- 
ance that is afforded in ILS approaches. 

Significant main effects for Geography, F( 1, 22) = 
4.83, /?<0.04, and for Final Items, F( 1, 22) = 6.13, 
p < 0.05 were also found, but in these cases more infor- 
mation was obtained in the ILS condition than in the 
VOR condition. A primary reason for this result might be 
that some elements in the category are only pertinent to 
ILS approaches (e.g., glideslope) or that flying an ILS 
approach afforded the crew more free time to look at 
this information (e.g., out-the-window validation of 
obstructions). 


18 



Weather , ; Statistically significant main effects for 
weather (adverse versus good) were observed for several 
of the information types. For Visibility Requirement, 
Missed Approach, Navigation, and Final Items, signifi- 
cantly more information of each type was gathered dur- 
ing adverse weather conditions than during good weather 
conditions. In addition to the simple need for more infor- 
mation when they are planning flights in adverse 
weather, pilots need to increase their monitoring of the 
fixes and target altitudes (Navigation) in adverse weather 
conditions to determine whether they are legal (Visibility 
Requirement), and if not, to identify the alternative 
(Missed Approach). 

More Obstructions information was gathered during 
good weather conditions than during adverse conditions, 
F(l, 22) = 5.04, p<0.05. This main effect was oppo- 
site of what was expected but may be attributable to the 
transfer of related tasks to ATC. Many pilots noted in 
their debriefings that obstruction avoidance became 
ATC’s primary responsibility in adverse weather 
conditions. 

Phase . As pointed out in the descriptive section 
above, the greatest amount of information was collected 
during the preapproach phase of flight. When analyzed 
statistically for each category, the above result was 
confirmed. For all similarity categories, except for Navi- 
gation, Obstructions, Visibility Requirement, and Geo- 
graphy significantly more information was acquired dur- 
ing the preapproach phase of flight. While no significant 
differences were found for the four information catego- 
ries listed above, only Obstructions and Geography dem- 
onstrated a pattern in which more information was 
collected during the approach phase than during the pre- 
approach phase. Again, this difference appears to be 
explained by the planning versus monitoring paradigm, 
in which Obstruction and Geography acquisitions in- 
crease during the approach while the progress of the 
approach is being monitored. 

Interactions. In addition to the main effects dis- 
cussed above, there were some statistically significant 
interactions of interest. While phase was not statistically 
significant for Navigation category acquisitions, a statis- 
tically significant crew-member-by-phase interaction 
emerged. While the PF’s acquisitions of Navigation 
information decreased from preapproach to approach, the 
PNF’s acquisition increased. Since the number of PF fly- 
ing tasks increases during the approach, the PF is accus- 
tomed to obtaining needed Navigation information from 
the PNF. This task allocation also confirms the planning 
and monitoring model of the approach and the interpreta- 
tion of the Navigation category. While both pilots are 
involved with planning during the preapproach, it is the 


PNF’s primary responsibility to monitor progress during 
the approach (and also to be a memory aid for the PF). 

For Navigation Aids information, a significant 
precision-by-weather interaction emerged. When pilots 
are flying a VOR approach in bad weather, the number 
of Navigation Aids acquisitions increases, whereas the 
number of acquisitions decreases while they are flying 
ILS in adverse weather. This pattern of acquisitions may 
be attributed to the use of items contained in this cate- 
gory for validation during approaches. The need to vali- 
date the current state increases in adverse weather 
because of the inability to make visual checks. One form 
of validation is to check that the radios are properly tuned 
to ascertain the correctness of the current flight profile. 
The possible reason that these checks decreased in bad 
weather for ILS while increasing for VOR is that positive 
glideslope and localizer signals on the flight deck help 
ascertain that the Navigation Aids are properly tuned 
when flying ILS, whereas checking the VOR signal can 
only be done against what is printed on the chart. 

A statistically significant interaction of phase by 
crew member emerged for Obstructions items. This 
interaction showed that the PNF acquired more informa- 
tion about Obstructions during the approach versus the 
preapproach, whereas the PF acquired more of this infor- 
mation during the preapproach. This explanation again 
seems to lie in the difference in task allocation among the 
crew members. The PF uses Obstructions information 
primarily for planning, whereas the PNF’s primary use 
of Obstructions information is in monitoring flight 
progress. 

The interaction between precision and weather when 
acquiring Missed Approach information is also of inter- 
est. While acquisitions of this information in good 
weather were somewhat constant for both VOR and ILS 
approaches, there were significantly more acquisitions of 
Missed Approach information when flying VOR 
approaches in adverse weather than when flying ILS 
approaches in adverse weather. An explanation might be 
that the crews had more trust in the avionics provided 
during ILS approaches to help them obtain a position that 
would prevent a missed approach. 

Relationship Between Pilots’ Models and 

Acquisition of Approach Information 

One of the primary goals of this study was to assess 
relationships between the pilots’ cognitive models and 
how pilots acquire information. Relationships between 
similarity categories and acquisitions were evident in the 
ANOVA results described above. If further results show 
that pilots acquire information relative to their cognitive 
models, empirically substantiated guidelines can be 
developed that use these models. This section addresses 
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Table 9. Preapproach Clustered Acquisitions 


Category 

Average 
items chart 

Selection 

probability 

Expected 

acquisition 

Actual 

acquisition 

Category 

selection 

probability 

Expected 

category 

acquisition 

Actual 

category 

acquisition 

Between 

category 

acquisition 

Navigation aids 

25.00 

0.130 

901 

1612 

0.017 

117 

740 

511 

Visibility requirement 

13.00 

0.068 

469 

220 


32 

106 

55 

Other runway 

13.75 

0.072 

496 

125 

0.005 

36 

47 

50 

Final items 

6.50 

0.034 

234 

56 


8 

4 

48 

Missed approach 

13.00 

0.068 

469 


0.005 

32 

141 

189 

Navigation 

59.25 

0.309 

2136 

2892 

0.095 

660 

1718 

461 

Obstructions 

31.50 

0.164 

1136 

636 


186 

207 

321 

Validation 

14.25 

0.074 

514 

948 


38 

594 

150 

Geography 

6.75 

0.035 

243 

18 

0.001 

9 

6 

9 

Communications 

9.00 

0.047 

325 

10 

0.002 

15 

5 

3 

Total 

192.00 

1.000 

6923 

6923 

0.163 

1132 

3569 

1797 


Table 10. Approach Clustered Acquisitions 


Category 

Average 
items chart 

Selection 

probability 

Expected 

acquisition 

Actual 

acquisition 

Category 

selection 

probability 

Expected 

category 

acquisition 

Actual 

category 

acquisition 

Between 

category 

acquisition 

Navigation aids 

25.00 

■EIE3H 

602 

899 

0.017 

78 

333 

378 

Visibility requirement 

13.00 

0.068 

313 

19 

0.005 

21 

9 

4 

Other runway 

13.75 

WES3M 

331 

28 

0.005 

24 

4 

20 

Final items 

6.50 

0.034 

157 

24 


5 

6 

15 

Missed approach 

13.00 

0.068 


162 

0.005 

21 

33 

102 

Navigation 

59.25 

0.309 

1428 

2593 

0.095 

441 

1597 

372 

Obstructions 

31.50 

0.164 

759 

683 

0.027 

125 

198 

377 

Validation 

14.25 

0.074 

343 

106 


25 

21 

70 

Geography 

6.75 

0.035 

163 

21 


6 

7 

8 

Communications 

9.00 

0.047 

217 

92 


10 

30 

39 

Total 

192.00 

1.000 

4627 

4627 

0.163 

756 

2238 

1385 


the further relationships between similarity categories 
and information acquisitions, information rankings and 
information acquisitions, and items recalled with infor- 
mation acquisitions, as well as with both the categoriza- 
tion and prioritization models. 

Similarity Categories and Information Acquisition 

One hypothesis of interest in this study was whether 
the pilots’ cognitive model of approach chart information 
similarity influences their acquisition of information 
from the approach chart. For instance, if tasks are sup- 
ported by similar items of information, then a dispropor- 
tionate number of acquisitions should be seen for some 
categories (i.e., those supporting the dominant tasks). A 
counterhypothesis would be that items on the approach 
chart are all equally likely to be selected and that the cat- 
egories that are composed of the most items will be 
selected most often. 


To test whether the categories are reflective of the 
information acquisitions, data were first compiled that 
concern the average number of chart items for each of the 
10 similarity categories, the probability of selection 
based on the number of items in a category, expected 
acquisitions (probability multiplied by total acquisi- 
tions), and the actual acquisitions from a category. These 
data are shown in table 9 for the preapproach phase and 
in table 10 for the approach phase. 

For the preapproach phase, a chi-square (% 2 ) test was 
performed to compare the expected versus the actual 
number of acquisitions from each of the 10 categories. A 
chi-square test is used to measure agreement or disagree- 
ment. If the observed values differ appreciably from 
the expected, then the chi-square value will be large. 
When the chi-square value is large, the hypothesis of 
independence is rejected. For the expected versus the 
actual number of acquisitions of the 10 categories, 
X 2 (9) = 2482.4, p < 0.001, which indicates that pilots 
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Table 11. Transitions Between Categories During Preapproach 



Navigation 

aids 

Visibility 

requirement 

Other 

runway 

Final 

items 

Missed 

approach 

Navigation 

Obstrs. 

Validation 

Geog. 

Comm. 

Navigation aids 

0.46 

0.00 

0.00 

0.00 

0.05 

0.38 

0.04 

0.07 

0.00 

0.00 

Visibility requirement 

0.02 

0.48 

0.00 

0.01 

0.20 

0.19 

0.01 

0.08 

0.00 

0.00 

Other runway 

0.05 

0.01 

0.41 

0.12 

0.08 

0.05 

0.01 

0.27 

0.01 

0.00 

Final items 

0.13 

0.00 

0.24 

0.07 

0.22 

0.13 

0.00 

0.17 

0.04 

0.00 

Missed approach 

0.21 

0.03 

0.05 

0.04 

0.36 

0.22 

0.04 

0.06 

0.01 

0.00 

Navigation 

0.19 

0.03 

0.00 

0.00 

0.04 

0.60 

0.10 

0.03 

0.00 

0.00 

Obstructions 

0.07 

0.00 

0.00 

0.00 

0.01 

0.52 

0.33 

0.05 

0.00 

0.00 

Validation 

0.16 

0.01 

0.03 

0.01 


0.09 


i 0.63 

0.00 

0.00 

Geography 

0.11 

0.06 

0.06 

0.00 

0.00 

0.11 

0.11 

0.22 

0.33 

0.00 

Communications 

0.10 

0.00 

0.00 

0.00 




0.10 

0.00 

0.50 


Table 12. Transitions Between Categories During Approach 



Navigation 

aids 

Visibility 

requirement 

Other 

runway 

Final 

items 

Missed 

approach 

Navigation 

Obstrs. 

Validation 

Geog. 

— 

Comm. 

Navigation aids 

0.37 

0.00 

0.00 

0.00 



0.06 

0.02 


0.01 

Visibility requirement 

0.00 

0.47 

0.00 

0.00 

0.16 

0.32 | 

0.00 

0.05 

0.00 

0.00 

Other runway 

0.07 

0.00 

0.14 

0.11 

0.11 

0.11 

0.04 

0.04 

0.00 

0.39 

Final items 

0.04 

0.00 

0.17 

0.26 


0.04 

0.04 

0.17 

0.00 

0.09 

Missed approach 

0.19 

0.01 

0.02 

0.02 

0.27 

0.36 

0.08 

0.04 

0.00 

0.02 

Navigation 

0.17 

0.00 

0.00 

0.00 

0.03 

0.63 

0.15 

0.01 

0.00 

0.01 

Obstructions 

0.09 

0.00 

0.00 

0.00 


0.58 

0.29 

0.02 

0.00 

0.01 

Validation 

0.21 

0.02 

0.01 

0.05 

0.04 

0.34 

0.11 

0.21 

0.00 

0.02 

Geography 

0.10 

0.00 

0.00 

0.05 


0.10 

0.24 

0.10 

0.33 

0.05 

Communications 

0.12 

0.00 

0.13 

0.00 


0.30 

0.08 

0.02 

0.00 

0.33 


were not equally likely to select from each of the derived 
categories. Since the data showed that the pilots were not 
randomly selecting from the categories, there is support 
for the notion that each category contains related ele- 
ments (i.e., supporting common tasks). 

Probabilities were also calculated and compared to 
actual values for the flying portion of the approach. 
These values are given in table 10. As in the preapproach 
analysis, a chi-square test was performed on the expected 
versus the actual number of acquisitions from each of the 
10 categories. This test yielded a (9) = 2203.2, 
p< 0.005, again demonstrating that pilots were not 
choosing equally from among all 1 0 categories. 

Also shown in tables 9 and 1 0 are the category selec- 
tion probabilities, which indicate the probability that 
there are two or more successive selections from a given 
category. These probabilities were again based on the 
number of items that comprise a category and therefore 
on the hypothesis that all items were equally likely to be 
selected. The probability of two succeeding selections 
within the same category would be the probability that 
the category would be selected multiplied by the proba- 


bility of its being selected. For example, the probability 
of Navigation Aids being selected is 0.130 (i.e., 25/192), 
and the probability of two Navigation Aids selections 
being made in succession is 0.017 (i.e., 0.130x0.130). 
With these probability data, it was then possible to calcu- 
late an expected number of clustered acquisitions (i.e., 
two or more acquisitions in succession from a category) 
and compare the expected number to actual clustered 
acquisitions. As tables 9 and 10 show for both the pre- 
approach and approach segments, the number of items in 
a category was a poor indicator of clustered acquisition; 
that is, acquisitions were determined more by similarity 
category than by expectation as determined by chance 
selection. 

Tables 11 and 12 show the transition frequencies 
between categories for the preapproach and approach 
segments, respectively. The largest frequencies for each 
row are in bold print. For example, in the preapproach 
the pilots looked at Navigation Aids in succession 
46 percent of the time and went from Navigation Aids to 
navigation information 38 percent of the time. For the 
preapproach segment, only Final Items and Obstructions 
had a higher between-category transition rate than 
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within-category transition rate. The within-category tran- 
sitions were second highest for Obstructions. 

For the flying portion of the approach, five catego- 
ries showed higher between-category transitions than 
within-category transitions (table 12). However, in all 
five cases (Navigation Aids to Navigation, Missed 
Approach to Navigation, Validation to Navigation, Other 
Runway to Communications, and Obstructions to Navi- 
gation) the within-category transitions were second in 
rank. The remaining five transitions were highest for the 
within-category. 

These transition statistics provide additional evi- 
dence that the identified categories contain functionally 
related information. The presence of functionally related 
information is evident because the highest, or second 
highest transition probabilities occur along the diagonals. 
This transition pattern indicates that it is more likely that 
a subject’s next information request will be from within 
the same category than from another category. The above 
result is consistent with the notion that the 10 categories 
contain similar information. Once a task is initiated, a 
pilot will probably continue selecting similar information 
to achieve that task’s goal. It is also plausible that the 
items required for completion of the task will come from 
the same general information category. The current 
results are consistent with and suggest this notion. Con- 
trast this result with a situation in which each category 
contains unrelated information. When there is unrelated 
information, the transition matrix would show all cells 
being of equal frequency. 

Information Rankings and Information 

Acquisitions 

Recall that each subject was asked to rank the 
approach chart information elements in terms of their 
perceived importance. Ranking was done under two con- 
ditions. The first was without regard to any particular sit- 
uation (context free) while the second was in the context 
of adverse weather (e.g., reduced minimums). The rela- 
tionship between these orderings and the pilot’s acquisi- 
tion of information from the approach chart was of 
interest. To examine this relationship the rankings for 
each of the two conditions were averaged across all sub- 
jects. These average rankings were then correlated with 
the total number of acquisitions provided in table 4. 
When the rankings were correlated with acquisition 
behavior, the correlations were quite low. The correlation 
between acquisitions and the context-free ranking was 
0.274, and the correlation between acquisitions and the 
adverse weather ranking was 0.259. At first these results 
may seem surprising, yet there is at least one potential 
explanation for the low correlations. If certain forms of 
information are extremely important, it is likely that the 


pilot will memorize them, write them down, or in the 
case of frequencies, enter them immediately after re- 
questing them. Thus, the assumption that higher impor- 
tance requires more acquisitions may be incorrect (as 
suggested by these data). 

Information Recall 

The information recall section pertains to the recall 
test that the subjects took concerning the information 
items on the approach chart of their last simulation flight. 
Statistics that concern the recalled information, how that 
information relates to other cognitive processes, and how 
it relates to acquisitions are discussed. 

Descriptive analyses. After the simulation portion of 
the study, subjects completed a “recall” quiz. On a paper 
copy of the grayed (illegible) chart for the last approach 
that was flown, each subject wrote all the information 
that he could recall. Table 13 gives a summary of the 
pilots’ statistics for this test. On average, subjects filled 
in 25.62 items, which accounted for 13.31 percent of the 
total number of items on the chart. The subjects correctly 
recalled 81.35 percent of the 13.31 percent of the items 
that they filled in. 

Table 14 lists only the elements correctly recalled by 
the subjects. The elements are arranged in the table 
according to the recall frequency. For instance, proce- 
dure name was most often recalled, followed by Visibil- 
ity Requirement, and fix name at FAF. Elements that are 
absent from the table never were recalled correctly by 
any of the subjects. 

The data in table 15 have been ordered by the num- 
ber of items in a similarity category (defined above) 
which were correctly recalled (from the most to the 
least). For instance, the Navigation category had the most 
items recalled (220) and the Other Runways category had 
the least (7). To test whether these numbers were just an 
artifact of the number of chart items that comprise a cate- 
gory, an expected value (i.e., expected recall) was calcu- 
lated for each category by multiplying the ratio of 
average chart items in a category by the average total 
items on a chart by the total number of items recalled. 
The last column presents the difference between the 
actual and expected recall. 

A chi-square test was performed to compare 
expected and actual recall of items. The value for 
table 15 data, y} (9) = 180.17, is significant at the 
1 -percent level and indicates that the pilots are not recall- 
ing information from within each of the categories with 
equal frequency. These data may suggest that the infor- 
mation that is contained within some of the categories is 
more critical to memorize, thereby leading to the in- 
creased recall. An equally possible explanation may be 
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Table 13. Information Recall Statistics 


Pilot 

No. tried 

Percent tried 

Percent correct 

Pilot 

No. tried 

Percent tried 

Percent correct 

1 

25 

12.32 

76.00 

14 

17 

9.29 

82.35 

2 

38 

18.72 

76.32 

15 

31 

15.27 

70.97 

3 

41 

22.28 

85.37 

16 

16 

7.88 

93.75 

4 

29 

15.76 

79.31 

17 

30 

14.78 

83.33 

5 

41 

22.40 

85.37 

18 

30 

14.78 

80.00 

6 

18 

9.84 

72.22 

19 

19 

10.33 

68.42 

7 

41 

20.20 

85.37 

20 

21 

11.41 

80.95 

8 

24 

11.82 

91.67 

21 

28 

15.30 

89.29 

9 

19 

9.36 

84.21 

22 

21 

11.48 

85.71 

10 

18 

8.87 

61.11 

23 

22 

10.84 

68.18 

11 

9 

4.89 

77.78 

24 

28 

13.79 

82.14 

12 

18 

9.78 

94.44 

25 

15 

8.15 

86.67 

13 

15 

8.20 

80.00 

26 

52 

28.26 

94.23 

Average 

25.62 

13.31 

81.35 






Table 14. Frequency of Elements Correctly Recalled 


Information element 

Recalled 

Information element 

Recalled 

Information element 

Recalled 

Procedure name 

54 | 

Chart index number 

10 

Glideslope angle 

3 

Visibility requirement 

37 

FAF intercept alt. (MSL) 

10 

GS intercept alt. (MSL) 

3 

Fix name at FAF 

27 

ILS identifier 

10 

Proc. turn dist. limit 

3 

Inbound course 

25 

Location of MAP 

10 

Tower freq. 

3 

Decision altitude (AGL) 

23 

Time from FAF to MAP 

id 

Airport name 

2 

Feeder route radial 

23 

VOR name 

10 

FAF intercept alt. (AGL) 

2 

Fix name 

22 

Fix name at IAF 

9 

MSA minimum alt. 

2 

Distance (non-DME) 

20 

ILS loc. magnetic course 

9 

MSA ref. point 

2 

VOR identifier 

20 

Outbound course 

9 

Rate of descent 

2 

Airport elevation 

19 

Decision altitude (MSL) 

7 

TDZE parallel runway 

2 

Missed approach proc. 

19 

Primary fac-freq. 

7 

AT1S arrival freq. 

1 

Min. safe alt. on AP 

18 

Approach freq. 

5 

Latitude-longitude grid 

1 

DME distance 

15 

Effective date 

5 

Lead radial 

1 

VOR freq. 

15 

ILS localizer freq. 

5 

Magnetic variation 

1 

Geographic name 

14 

Holding pattern course 

4 

Runway length 

1 

Obstruction 

12 

MAP (nonprecision) 

4 

Step-down alt. (AGL) 

1 

Step-down alt. (MSL) 

11 

Other runway numbers 

4 

Threshold cross height 

1 

Chart date 

10 

TDZE target runway 

4 




Table 15. Categories and Information Recall Statistics 


Category 

Types in category 

Average chart items 

Items recalled 

Expected recall 

Difference 

Navigation 

21 

59.25 

220 

169 

51 

Validation 

9 

14.25 

100 

41 

59 

Navigation aids 

12 

25.00 

67 

71 


Missed approach 

6 

13.00 

47 

37 

10 

Visibility requirement 

1 

13.00 

37 

37 

0 

Obstructions 

6 

31.50 

34 

90 

-56 

Geography , 

4 

6.75 

16 

19 

-3 

1 

Final items 

5 

6.50 

10 

19 

-9 

Communications 

7 

9.00 

9 

26 

-17 

Other runway 

4 

13.75 

7 

39 

-32 

Total 

75 

192.00 

547 

547 
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that some items just do not promote or require memoriza- 
tion, such as Communications, which are annunciated by 
tuning the instruments and therefore are not memorized. 
Overall, values indicate that the number of items repre- 
sented in a category was not a good indicator of how 
often items in a category would be recalled and that the 
category memberships themselves might influence which 
items are memorized. 

The perceived importance of approach information 
was a poor predictor of which items pilots memorize. 
The correlation between element types that were cor- 
rectly recalled versus their average importance rankings 
was 0.440, indicating that importance played a very 
small role in determining which items pilots commit to 
memory. When the number of acquisitions was normal- 
ized by the number of chart items of that type, this corre- 
lation dropped to 0.382. 

Items remembered and information acquisition . 

This section addresses whether a relationship exists 
between the number of information acquisitions and the 
subjects’ recall of those items. Since the selected infor- 
mation disappeared from the screen after 15 sec, it was 
often necessary for subjects to repeatedly acquire the 
information. It would seem logical to assume that the 
more times the information was acquired the more likely 
it would be for the pilot to recall it later. To examine the 
likelihood of recalling often repeated information, the 
number of times each piece of information from the 
approach chart was acquired was paired with the number 
of times that information was correctly recalled. A corre- 
lation coefficient was calculated between these variables, 
yielding r = 0.618 (r is the correlation coefficient). This 
correlation demonstrates that the recalled items were 
selected more often than those that were not recalled. 

It would also seem logical that the more recently an 
item was acquired (i.e., nearer the end of the flight) the 
more likely it would be for the pilot to recall it. Thus, the 
average time of each item’s last access (in percentage of 
overall time) was compared to whether or not it was cor- 
rectly recalled. This analysis showed that items that were 
correctly recalled were accessed more recently than 
items that were not recalled. 

Memorization questions summary . After the sub- 
jects took the recall test, they completed two questions 
concerning the memorization of approach chart informa- 
tion. The first question asked whether, under actual fly- 
ing conditions, pilots make a conscious effort to 
memorize approach chart information during the pre- 
approach briefing so that they will not have to acquire it 
again during descent and approach. Eleven said that they 
did try to memorize items but 14 did not. (One subject 
did not answer these two questions.) The second question 


was whether pilots made a conscious effort during the 
simulation flights to memorize approach chart informa- 
tion during the predescent briefing so that they did not 
have to acquire it again during descent and approach. 
Fourteen said that they did make a conscious effort to 
memorize and 1 1 said they did not. Subjects also sup- 
plied a list of elements that they felt they memorized. 
The correlation (0.330) of the elements the pilots thought 
they memorized to what was correctly recalled indicated 
that the pilots have a poor understanding of what they 
actually commit to memory. 9 A possible explanation for 
this difference might be that pilots felt that important ele- 
ments are memorized, but an earlier analysis showed that 
an item’s perceived importance was not directly related 
to its likelihood of being recalled. 

Information Requirements 

Past information requirements analyses (Boeing 
1991a, 1991b; Hansman and Mykityshyn 1990; 

Mykityshyn and Hansman 1990, 1991) of approach chart 
use help identify the information that is necessary for 
safe descent from an en route altitude to the landing alti- 
tude of a destination runway. These studies concentrated 
primarily on observable tasks (e.g., descent) and not on 
the cognitive tasks (e.g., planning), which are highly rep- 
resented when managing approach information. Given 
the dominance of cognitive tasks when one is managing 
information, the study described in this paper further 
investigated the information requirements of approach 
chart use and emphasized cognitive task and actual use. 
Results of this study are intended to augment previous 
studies that identified the information that is required by 
identifying what information is acquired , when it is 
acquired, and what presentation concepts might facilitate 
its efficient use by better matching the pilots’ mental 
model of the information with its presentation. 

Categories of Approach Chart Information and 

Information Requirements 

Knowing how pilots mentally organize approach 
chart information provides cartographers and flight-deck 
system designers with knowledge that is useful in their 
endeavors for efficient interfaces. Having information 
formatted in a way that matches the users’ mental models 
improves the users’ efficiency when they are working 
with that information (Clay 1993; Woods 1985). This 
study indicated that pilots use 10 primary categories 
to organize approach chart information: Communica- 
tions, Geography, Validation, Obstructions, Navigation, 

9 The inability of individuals to articulate how they do things 
mentally spans all domains. This inability is one reason why 
cognitive models are explored using techniques other than just 
interviews. 
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Missed Approach, Final Items, Other Runways, Visibil- 
ity Requirement, and Navigation Aids. These similarity 
categories appear to underlie many of the cognitive tasks 
that are invoked by pilots and were highly reflective in 
their acquisition of approach information. Presenting 
information according to these classifications should 
facilitate pilots’ use of approach chart information by not 
requiring them to expend additional cognitive effort 
organizing the information prior to cataloging and using 
it. If information is presented according to the categories 
that are identified in this study, pilots should be able to 
extract and use the information easily by recognizing it 
in the contexts that support the higher level cognitive 
tasks. Please refer to appendix F for a brief elaboration 
on how these categories might be used to organize 
approach charts and the electronic presentation of 
approach information. 

Importance of Approach Chart Information and 

Information Requirements 

A review of the pilots’ perception of the importance 
of approach chart information indicated that there was 
stability across the contexts used in this study. Stability 
across contexts is critical if design guidelines are to be 
derived from these data. When combined with the actual 
use of information, perceived importance of information 
can help determine which information to highlight or 
duplicate in its presentation concept. These guidelines 
can be derived by the average ranking of the individual 
element types (e.g., ILS localizer magnetic course) or by 
the relative order of similarity categories in the average 
rankings. By examining the relative order of similarity 
categories, it was observed that many of the element 
types that are identified as highly important come from 
the similarity categories of Visibility Requirement (“Are 
we legal to fly this approach?”) and chart Validation 
(“Do we have the correct and same approach?”). Also 
note-worthy was that Geography and Other Runways 
information occupied the “least important” positions. 

Acquisition of Approach Chart Information and 

Information Requirements 

Many design guidelines can be derived from observ- 
ing the actual acquisition of information. Two primary 
findings that concern the acquisition of approach chart 
information were that similarity categories were highly 
represented in the acquisitions and that the requirements 
(reflected by the acquisitions) changed as a function of 
flight phase. These findings suggest that the categories 
can be used to help define both the presentation formats 
and the tailoring of information (when the information is 
presented electronically). 


Figure 20 presents the data that were discussed ear- 
lier in a slightly different way. In figure 20 the acquisi- 
tions from the similarity categories are plotted as a 
function of flight phase. As can be seen, all the categories 
of information are used primarily during the preapproach 
phase. As discussed earlier, this result supports the the- 
ory that the primary use of the approach charts is for 
planning (versus monitoring). Acquisitions from the dif- 
ferent categories then vary somewhat for the remaining 
phases of the approach. These data on category usage can 
directly affect design guidelines, or emphasis can be 
placed on the acquisition of individual items instead of 
on the categories, as discussed earlier. 

Additional analyses showed that some categories 
were more likely to be accessed than others, independent 
of the number of elements that comprise the category 
(refer to tables 9 and 10). This likelihood again can affect 
both guidelines that concern the presentation formats 
(e.g., salience) and the tailoring of information. The tran- 
sition frequencies shown in tables 1 1 and 12 are another 
finding that would affect the dynamic tailoring of infor- 
mation. The transitions show that when a pilot acquires 
information from a given category, the next acquisition 
(i.e., the transition) is probably another element within 
the same category. 

Usually, the perceived importance of information 
dictates its presentation format for arranging choices 
such as the salience or repetition of information. How- 
ever, a primary reason for making a specific piece of 
information more salient or for repeating it is to make it 
easier to acquire. Analyses in this study showed that 
there was a very low correlation between importance and 
frequency of acquisition. This result suggests that neither 
frequency nor importance should be used by itself to dic- 
tate guidelines that concern the salience of information 
presentation formats. 

Conclusions 

Many tasks that are associated with using approach 
charts are cognitive, such as planning and monitoring. 
Clay found that pilots were required to do more tasks 
during the approach segments of flight and that because 
of this increase in cognitive workload, there is increased 
importance associated with the abilities of the pilot to 
obtain information quickly and accurately to support 
these tasks and remember the information long enough to 
apply it. To aid the pilot during this phase of flight, 
design efforts associated with the presentation of 
approach information should consider the pilots’ cogni- 
tive demands. The study reported in this paper, managing 
approach plate information study (MAPLIST), com- 
bined psychometric scaling techniques with a simulation 
task to provide cartographers and flight-deck system 
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designers with such information. This study provides 
quantifiable links between pilots’ cognitive representa- 
tions of approach information and their use of approach 
information. 

Categorization is a dominant cognitive process that 
supports cognitive tasks in all domains that involve infor- 
mation management. Categories of information are used 
by people to reduce the amount of information with 
which they work. For instance, pilots do not treat each 
item of information on the approach chart as a com- 
pletely independent element (an impossible task given 
the amount of information). Instead, pilots categorize 
(i.e., organize by similarity) the information, manage it 
through mental manipulations of the category, and then 
use the mental classification to access the individual item 
when it is needed. Through this process pilots are able to 
comfortably work with the approach chart by not being 
overwhelmed by the amount of information it contains. 

A primary finding in this study was that pilots appear 
to mentally organize approach chart information into 
10 primary categories: Communications, Geography, 
Validation, Obstructions, Navigation, Missed Approach, 
Final Items, Other Runways, Visibility Requirement, and 
Navigation Aids. These similarity categories were found 
to underlie many of the cognitive tasks invoked by pilots 
and were highly reflected in their acquisition of approach 
information. Given these findings, it is reasonable to 
believe that the use of these categories in chart design 
may facilitate pilots’ use of approach charts. Incorporat- 
ing the categories in the presentation formats could 
relieve the pilot of the additional cognitive effort that is 
associated with organizing the information prior to cata- 
loging and using it. When information is presented using 
the categories found here, pilots should be able to extract 
the information more easily by recognizing it in a context 
that supports their higher level cognitive tasks. Categori- 
zation is already being done by the pilots to prevent over- 
load when they are dealing with a large amount of 
information (such as that on an approach chart). Having 
the presentation coincide with this process should 
improve efficiency and decrease errors that are associ- 
ated with its use. Appendix F provides a brief elaboration 
on possible applications of these categories to approach 
chart organization and electronic presentation concepts. 

As stated previously, the similarity categories 
appeared significant in many of the analyses, thus giving 
more support for their use in presentation formats. For 
the average importance rankings of the information 
obtained from the pilots, information elements from 
within the same categories appear close together in the 
ranking data. For instance, elements that comprise Vali- 
dation and Visibility Requirement were given high 
importance rankings, while members of the Geography 


and Other Runways categories were given low priority 
ratings. A strong relation with categories allows the 
designer to use the relative order of either the similarity 
categories or individual elements within similarity cate- 
gories to decide presentation-format issues that concern 
priority (e.g., salience or repetition). 

It was also suggested that adverse weather did not 
affect the pilots’ perception of the importance of infor- 
mation. However, the temporal position within the 
approach did seem to affect the perceived importance of 
the information. That is, the importance of an informa- 
tion type seemed to depend on approach segment (i.e., 
preapproach, approach transition, intermediate, or final 
segment). This change in importance may be a reflection 
of the information requirements of the individual seg- 
ments and again provides design insight for dynamic tai- 
loring of information presentations. 

The acquisition data provide many insights for pre- 
sentation formats and for dynamically tailoring informa- 
tion. Results showed that a greater amount of in- 
formation was collected during the preapproach phase of 
flight. Acquisition counts showed that the information 
types that were acquired varied most with the approach 
segments (i.e., transition, intermediate, final, and 
missed). When they were analyzed statistically for each 
category, five categories were acquired significantly 
more in preapproach. For the remaining five categories, 
only Obstructions and Geography demonstrated a pattern 
in which more information was collected during the 
approach phase than during the preapproach phase. 
These findings were just a few of many that supported 
the planning versus monitoring task allocation paradigm 
that is associated with approach chart use. 

Additional analysis of variance (ANOVA) results 
(with the similarity categories as dependent measures) 
showed that the pilot not flying (PNF) acquired more 
information than the pilot flying (PF) for five of the cate- 
gories. The main effect of precision indicated that more 
Navigation, Obstructions, and Missed Approach infor- 
mation was obtained when pilots were flying very high 
frequency (VHF) omnidirectional range (VOR), while 
more information was acquired from the Geography and 
Final Items categories when they were flying instrument 
landing system (ILS). More information was also 
acquired during adverse weather scenarios of VOR 
approaches than of ILS approaches. While it was not 
immediately obvious, a significant main effect of 
weather showed that Obstructions information was 
acquired more in good weather. More intuitive were the 
significant increases in acquisitions from Visibility 
Requirement, Missed Approach, Navigation, and Final 
Items for adverse weather. 
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Further investigation into the relationship of similar- 
ity categories and the acquisition of approach informa- 
tion showed that category membership provided a good 
indication of selection probability. Certain categories of 
information were selected more often than others, inde- 
pendent of the number of items that comprise the cate- 
gory. Analyses also showed high within-category 
transitions for each of the categories. These transition 
relationships provide additional evidence that the identi- 
fied categories contain functionally related information, 
since similar items are probably necessary to support a 
task. The transition relations also provide additional 
insight into how information might be tailored if pre- 
sented electronically. 

When testing subjects for information that was 
recalled from the last approach, the category membership 
of an information item was a good indication of whether 
it would be remembered. This finding was independent 
of the number of items that comprise the category. Both 
Validation and Navigation information were recalled 
well whereas members of Obstructions, Communications 
and Other Runways were not recalled well. The correla- 
tion of an element’s recall frequency with its average 
importance ranking showed that importance was not a 
strong indicator of which information would be recalled. 
However, more exposure to an item increased the likeli- 
hood of its recall. A fairly high correlation of acquisition 


frequency to recall frequency showed that the items that 
were remembered had been accessed more than those 
items that were not remembered. Another finding was 
that items that were recalled correctly were on average 
accessed more recently than those that were not recalled 
or that were recalled incorrectly. 

Results of this study are intended to augment previ- 
ous work that identified what information was required 
by identifying what information is acquired , when it is 
acquired, and the mental models that pilots have of this 
information. Augmentation of this type should help car- 
tographers and flight-deck system designers devise pre- 
sentation concepts that facilitate efficient use by better 
matching the pilots’ cognitive models and use of the 
information with its presentation. For instance, the simi- 
larity categories were salient in the pilots’ use of 
approach charts. Therefore, using these underlying cate- 
gories of information in designing presentation formats 
might ease related pilot workload by not requiring the 
pilot to organize the information to the same extent prior 
to using it. 


NASA Langley Research Center 
Hampton, VA 23681-0001 
October 27, 1995 
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Appendix A positioning of some of the information and symbols that 

are used. (See fig. Al, taken in most part from Jeppesen 
Approach Chart Formats Sanderson, 1993, pp. 3-6.) 

Instrument approach charts are currently published 

by Jeppesen Sanderson, Inc., (hereafter referred to as Both formats have a heading section, plan view, pro- 

Jeppesen) and the National Oceanic and Atmospheric file view, landing minimums, missed approach instruc- 

Administration (NOAA) National Ocean Service (here- tion, and a missed approach point (MAP) table. The NOS 

after referred to as NOS). Both Jeppesen and NOS charts charts also have two additional sections: one has notes 

provide the pilot with all the information that is needed and cautions (that are placed in either the plan view or 

for a given runway. While both charts share more com- profile view of Jeppesen charts), and the other shows the 

monalties than differences, there are differences in the airport diagram (separate chart with Jeppesen format). 



Figure Al. Approach chart layouts. 
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Appendix B 

Preapproach and Approach Time Profiles and 
z Scores 

The z score is the number of standard deviations that 
a given data point is above or below the mean. A positive 
z score represents an above-the-mean score, and a nega- 
tive z score represents a finding that is below the mean. 
Because of a procedural error during scenario 2 for 


crew 10, the subsequent z score for their total time 
was 3.443, which was well outside the range that is 
acceptable for this study (-2 to 2). All others scores were 
within the range -1.953 through 1 .864 and were retained 
for analysis. 

In tables B1 through B5, phase 0 refers to the 
approach transition segment. Phase 1 refers to the inter- 
mediate segment. Phase 2 refers to the final approach 
segment. Phase 3 is the missed approach segment. 


Table Bl. Scenario 1 Simulation Statistics 



Time, sec 

z score 



Crew 

Phase 0 

Phase 1 

Phase 2 

Phase 3 

Total 

Phase 0 

Phase 1 

Phase 2 

Phase 3 

Total 

1 

578.500 

102.400 

158.700 







-0 4060 

2 



148.500 


814.100 





-1.1763 

3 

604.100 

87.100 

158.700 



0.5293 | 




-0.0948 

4 

640.000 

97.300 

168.900 


906.200 



0.6610 


1.6060 

5 



163.800 


855.000 





0.0593 

6 



102.400 

35.800 

788.400 





-1.9527 

7 

552.900 

138.300 

158.700 







-0.0948 

8 



158.700 


844.800 

0.3007 

-0.8659 

0.2206 


-0.2489 

9 



174.100 


885.700 

0.7579 

0.0449 

0.8856 


0.9867 

10 



174.100 ^ 


865.300 

-0.6182 

0.6580 

0.8856 


0.3704 

11 

604.100 

133.200 

174.100 


911.400 





1.7631 

12 

578.500 

107.600 

102.400 

35.800 

824.300 



-2.2106 


-0.8682 

10b 

609.200 

92.100 

153.600 







0.0562 

Average 3 

592.292 

101.646 

153.592 








RIPISISP 



16.535 









22.309 

16.799 

23.158 









a Mean 

b Average of absolute deviations from the mean 
c Standard deviation from the mean 
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Table B2. Scenario 2 Simulation Statistics 


Crew 

— 

Time, sec 

z score 

Phase 0 

Phase 1 

Phase 2 

Phase 3 

Total 

Phase 0 



Phase 3 

Total 

1 

578.500 

82.000 

163.800 


824.300 

-0.5627 




-0.3821 

2 

599.000 

87.100 

153.600 







-0.3275 

3 

599.000 

87.100 

158.700 


844.800 

0.1688 

-0.3368 

-0.3019 


-0.3094 

4 

640.000 

97.300 

168.900 







-0.0918 

5 

619.500 

92.200 

158.700 





-0.3019 


-0.2187 

6 

558.000 

92.100 

138.300 


788.400 

-1.2942 

-0.3164 

-0.9523 


-0.5093 

7 

558.000 

128.000 

158.700 


844.700 

-1.2942 

-0.1700 

-0.3019 


-0.3098 

8 

614.400 

92.100 

153.600 


860.100 

0.7184 

-0.3164 



-0.2552 

9 

634.800 




906.100 

1.4463 

-0.2744 

0.0233 


-0.0922 

10 

614.400 

1017.800 

271.400 


1903.600 

0.7184 

3.4598 

3.2912 


3.4432 

11 

568.300 

102.400 

179.200 


849.900 

-0.9267 

mgfSEm 

0.3517 


-0.2914 

12 


112.600 

163.900 


860.100 

-0.3807 

-0.2328 

-0.1361 


-0.2552 

10b 

558.000 


148.500 


819.100 

-1.2942 

£|VWj| 

-0.6271 


-0.4005 

Average 3 

594.269 

169.669 

168.169 


932.108 








130.482 

17.804 


149.460 






Stdevp c 

28.024 

245.140 

31.365 


282.152 







a Mean 

b Average of absolute deviations from the mean 
c Standard deviation from the mean 


Table B3. Scenario 2 Simulation Statistics Without Crew 10 


Crew 

Time, sec 

z score 

Phase 0 

Phase 1 

Phase 2 

Phase 3 

Total 


Phase 1 

Phase 2 

Phase 3 

Total 

1 

578.500 

82.000 

163.800 


824.300 

-0.5627 

-0.3576 

-0.1393 


-0.3821 

2 

599.000 

87.100 

153.600 


839.700 

0.1688 

-0.3368 

-0.4645 


-0.3275 

3 

599.000 

87.100 

158.700 


844.800 





mmsm 

4 


97.300 

168.900 


906.200 

1.6319 

-0.2952 

0.0233 


-0.0918 

5 

619.500 

92.200 

158.700 


870.400 

0.9003 

-0.3160 

-0.3019 


-0.2187 

6 

558.000 

92.100 

138.300 





-0.9523 


-0.5093 

7 

558.000 

12.800 

158.700 


844.700 






8 


92.100 

153.600 


860.100 

0.7184 

-0.3164 

-0.4645 


-0.2552 

9 

634.800 

102.400 

168.900 


906.100 

1.4463 

-0.2744 

0.0233 



10 

568.300 

102.400 

179.200 




-0.2744 

0.3517 


-0.2914 

11 

583.600 

112.600 

163.900 


860.100 

-0.3807 

-0.2328 

-0.1361 


-0.2552 

12 

558.000 

112.600 

148.500 


819.100 

-1.2942 




-0.4005 

Average 3 

592.592 

98.992 

159.567 


851.150 


MlillMM 

IttlillM 




25.192 

10.507 

7.811 


24.525 






Stdevp c 

28.534 

12.750 

10.184 


32.250 







a Mean 

b Average of absolute deviations from the mean 
c Standard de viation from the mean 
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Table B4. Scenario 3 Simulation Statistics 



Time, sec 

z score 


Crew 

Phase 0 

Phase 1 

Phase 2 

Phase 3 

Total 

Phase 0 

Phase 1 

Phase 2 

Phase 3 

Total 

1 

650.200 

51.200 

128.000 


829.400 

-0.4474 

-1.9251 



-0.4950 

2 



107.500 


844.800 

0.7640 




0.1502 

3 


56.300 

112.700 


814.100 

-0.6887 

-0.9669 

-0.5488 


-1.1360 

4 



122.800 


829.300 

-0.4474 

-0.9669 





-0.4992 

5 



133.100 


844.800 





0.1502 

6 


61.500 

128.000 


839.700 

-0.4474 

0.0101 

0.6481 


-0.0635 

7 



81.900 

5.200 

803.900 





-1.5633 

8 

670.700 

61.400 

122.800 


854.900 





0.5733 

9 

655.300 

71.700 

128.000 


855.000 

-0.2060 

1.9266 

0.6481 


0.5775 

10 



128.000 1 


885.700 

1.9754 




1.8636 

11 

686.000 

61.400 

122.900 


870.300 

1.2467 

-0.0087 

0.2491 


1.2185 

12 

675.800 H 

66.600 

1 17.800 


860.200 

0.7640 

0.9683 

-0.1498 


0.7953 

10b 

614.400 

66.500 

122.800 


803.700 

-2.1414 

0.9496 

0.24131 


-1.5716 



61.446 

119.715 


841.215 







17.143 

3.966 

9.071 


19.568 








5.322 

12.784 


23.870 







a Mean 

b Average of absolute deviations from the mean 
c Standard deviation from the mean 


Table B5. Scenario 4 Simulation Statistics 



Time, sec 

z score 


Crew 

Phase 0 

Phase 1 

Phase 2 

Phase 3 




Phase 2 

Phase 3 

Total 

1 



117.800 


768.000 


0.0278 

BiSUEfl 


-1.6397 

2 


66.500 

87.000 


793.500 

-0.5214 

0.7647 

-2.4555 


-1.0540 

3 

734.400 

40.800 

117.300 


892.500 

2.0108 

-1.0921 

-0.4463 


1.2201 

4 



122.900 


860.200 

0.3021 




0.4781 

5 

650.200 

76.800 

133.200 


860.200 

-0.2478 

1.5089 

0.6080 


0.4781 

6 



122.900 


809.000 

-1.0713 

0.7719 

-0.0750 


-0.6979 

7 

655.300 

35.900 

143.400 





1.2844 


-0.1099 

8 



112.600 


829.300 

0.0258 

0.0278 

-0.7580 


-0.2316 

9 

665.600 1 

66.500 

122.800 





-0.0816 


0.3564 

10 



128.000 


880.600 

1.1256 

-0.3407 

0.2632 


0.9467 

11 

675.800 

61.500 

122.900 


860.200 

0.4389 




0.4781 

12 



128.000 


762.800 

-1.3476 1 

-2.1902 

0.2632 


-1.7591 

10b 

696.300 

56.300 

153.600 




0.0278 

1.9607 1 


1.5347 


HESS!! 


124.031 


839.385 






!S5 

■ 

10.794 

10.161 


36.786 






Stdevp c 

37.279 

13.841 

15.081 


43.535 







a Mean 

b Average of absolute deviations from the mean 
c Standard deviation from the mean 
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Appendix C 

Debriefing Questionnaire and Responses 

Debriefing questions and summaries of responses 
are provided in this appendix. Comments have been 
edited; those that were not pertinent were deleted, and 
those that pertained specifically to differences between 
the MD80 and the DC-9 series 30 were deleted. Since all 
American Airline pilots made adjustments because of 
their use of above ground level (AGL) altitudes instead 
of mean sea level (MSL) altitudes, all related comments 
were also removed. 

The first question was how similar the simulator 
response and other general handling characteristics were 
to the aircraft that pilots were accustomed to. One pilot 
felt the response and other general handling characteris- 
tics were very similar, while 17 judged them similar and 
8 said they were not similar. Specific comments concern- 
ing this question have been paraphrased below: 

• Pitch and roll rates don’t feel similar, and fuel 
flow was not close. 

• The simulator was more sensitive (i.e., move 
the nose slightly and the vertical speed indica- 
tor and the (VSI) really jumped). 

• The simulator was more sensitive than aircraft 
in all phases of pitch, roll, and yaw. 

• The simulator was overly sensitive in pitch 
and roll. 

• The simulator was too sensitive in pitch with 
gear down. 

• The simulator was too sensitive to rudder 
inputs on landing rollout. 

The second question was how similar the onboard 
equipment-systems were to what the pilots are accus- 
tomed to. Three pilots judged them very similar, 1 8 said 
they were similar, and 5 said they were dissimilar. Com- 
ments specific to this question are presented below: 

• There are differences in flight director 
controls. 

• The altitude selector is different and in a dif- 
ferent location. 

The next question asked whether pilots needed to 
change their normal strategies in order to accomplish 
their tasks. In response to this question 9 pilots said they 
did change their normal strategies while 17 said they did 
not. Comments offered to this question were as follows: 

• There were different checklist procedures. 

• Approach chart information retrieval was 
more time consuming, thus reducing the 


amount of attention spent on flying the 
aircraft. 

• I memorized more to avoid using the mouse 
during a critical phase of flight. 

• I used only heading select and vertical speed 
due to different autopilot. 

The next series of questions that the pilots answered 
pertained to the acquisition device used in the simulator. 
The first question asked whether the position of the 
approach chart display generally satisfied pilots’ needs. 
In response to this question, 23 pilots said the position 
was generally satisfactory, whereas 2 said it was not 
(1 pilot did not answer this question). Comments specific 
to this question are listed below: 

• I prefer up and down eye movement instead of 
left to right head movement. 

• Excessive head movement could induce 
vertigo. 

• One must look 30 deg away from the instru- 
ments to see the monitor. 

The next question asked whether the positioning and 
operation of the cursor-control device was generally sat- 
isfactory to meet pilots’ needs. Twenty-five subjects 
answered yes (again, one pilot did not answer this 
question). 

The next series of questions addressed the approach 
information that was used in the simulator. The first one 
asked whether the information on the screen was easy to 
read. Twenty-one pilots said that it was easy to read and 
4 said it was not. Comments that are specific to the pre- 
sentation of the approach information in the simulator are 
compiled below: 

• The resolution could have been better. 

• Some areas, such as distance between fixes, 
were difficult to read. 

• Runway information was very small. 

• Some items were difficult to click on. 

• Some of the smaller crowded information was 
difficult to read. 

When asked whether the acquired information 
remained on the display long enough, 8 pilots said that 
display time was usually too short; 17 said it was usually 
just right; none said that it was too long (1 pilot did not 
answer). Additional comments indicated that there were 
times when items faded sooner than the pilots would 
have liked. 

Concerning the acquisition device, the pilots were 
asked whether they experienced difficulties using the 
cursor-control device to acquire information from the 
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display. In response to this question, 2 pilots said that 
they often experienced difficulties, 20 said that they 
rarely experienced difficulties, and 4 said that they never 
experienced difficulties. Specific comments are listed 
below: 

• Sometimes I couldn’t get information on 
screen because I was hurrying — tried to get 6 
or 7 items displayed at once and that caused 
me to make an error in point-shoot technique. 

• There was a learning curve. 

• I did have numerous miss clicks of informa- 
tion displayed in a group (i.e., the frequency 
box). 

When asked whether they had difficulties seeing the 
cursor on the screen, no pilots answered always or occa- 
sionally. Eleven pilots said that they rarely had difficul- 
ties, and 15 said that they never had problems. 

The pilots were also asked whether they recalled any 
instances in which they decided not to acquire informa- 
tion that they wanted because they were busy with the 
airplane. Ten said that they did recall such instances, 
whereas 16 said they did not. Comments pertaining to 
this question are listed below: 

• On final I finally learned to pre-position the 
cursor to the decision height position for quick 
access. 

• I often recall instances when I was too busy to 
acquire information because the time element 
was critical (i.e., course change, altitude 
change, or approach minimums). 

• Once in a while I was too busy during aircraft 
reconfiguration or altitude transition; I went 
back to it later. 

• Nearing an event like altitude capture or 
course capture, I deferred accessing informa- 
tion until after the event. 

• I just asked the pilot not flying to supply 
needed information. 

• Had I been able to simply look to acquire 
information, I would have. 

• During a critical period such as a turn and 
descent or intercept, I was too busy. 

• I was too busy on final approach. 

• I was too busy two or three times: during busi- 
est parts of approach (turns, descents). 

• I generally asked the PNF for any information 
I needed. 

Pilots were also asked whether the acquisition task 
changed the frequency with which they normally 


obtained information from an approach chart. Fifteen 
said that it did, whereas 1 1 said that it did not. The com- 
ments listed below give a good indication of how the task 
changed the frequency of acquisitions. 

• I started asking the PNF to retrieve and repeat 
information. 

• Since it took longer, I didn’t cross-check as 
frequently. 

• I had to memorize more information. 

• I tried to plan ahead more. 

• Information is generally gathered in a more all 
encompassing way (i.e., when checking for 
lead-in information, I would always note the 
radial, altitude, and distance when I looked for 
the radial). 

Pilots were also asked whether the different 
approaches appeared to be approximately equal in diffi- 
culty. Twenty-four of the pilots said they did appear 
equal in difficulty, and two said they did not. Comments 
pertaining to this question are listed below: 

• Short feeder routes made descent altitude 
more time-critical. 

• The task was complicated by large angle turns. 

• There were normal differences between preci- 
sion and nonprecision. 

• Two of the approaches were mirror images 
and some of my selections were to verify that 
instead of gathering information. 

The pilots were asked to exclude the acquisition task 
and tell how similar the tasks that they performed in the 
simulator cockpit were to those they would normally per- 
form as captain or first officer. Fifteen pilots said the 
tasks were very similar and 1 1 said they were similar. 
None said that the tasks were dissimilar. Specific com- 
ments are listed below: 

• You may warn the pilots to bring their normal 
checklists with them, as opposed to using the 
Northwest 1989 checklist. 

• Except for the outdated other airline checklist 
they were similar. 

The next question asked how adequate the informa- 
tion was that was provided to them by air traffic control 
(ATC). Fourteen said it was very adequate and 12 said it 
was adequate. None said that ATC support was inade- 
quate. Specific comments for this question are listed 
below: 

• The controller was very good. 

• If this was Hayes, Kansas, the level of support 
was about right, but for busier airports we 
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almost always receive vectors to the final 
approach course. 

Pilots were also asked whether the way approaches 
were flown in the experiment changed the way they used 
the approach chart from the way they would normally 
use it when flying the line. Five said it did change things, 
whereas 21 said it did not. Specific comments concern- 
ing this question are listed below: 

• Flying domestic routes in the U.S., I rarely fly 
an entire approach procedure from a feeder 
route. 


• Normally we are vectored to final. 

• I don’t memorize as much information. 

• Very seldom in line flying would you fly the 
whole approach. 

• We are almost always vectored to final. 

• Flying the whole approach with transitions 
meant that I used more information from the 
chart than I would normally use. 
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Appendix D 

Analysis of Variance (ANOVA) Summary Tables 

The following tables represent the analyses of variance that were done on each of the dependent variables. For each 
treatment effect or interaction in the tables, the sum of squares (SS), degrees of freedom (df), mean square (MS), F ratio, 
and significance level (p ) are reported. 


Table Dl. Communications Category ANOVA Summary Table 


Source 

df 

SS 

MS 

F 

P 

Between subjects 


Crew member (CM) 

1 

18.13 

18.13 

3.71 

0.07 

Error 

22 

107.45 

4.89 



Within subjects 

Precision (T) 

1 

0.13 

0.13 

0.15 

0.71 

CM XT 

1 

0.13 

0.13 

0.15 

0.71 

Error 

22 

19.37 

0.88 



Weather (WX) 

1 

0.01 

0.01 

0.01 

0,94 

CMxWX 

1 

0.63 

0.63 

0.75 

0.40 

Error 

22 

18.49 

0.84 



Phase (P) 

1 

32.51 

32.51 

7.02 

0.02 

CM x P 

1 

26.26 

26.26 

5.67 

0.03 

Error 

22 

101.87 

4.63 



Tx WX 

1 

1.51 

1.51 

3.21 

0.09 

CM x [T x WX1 

1 

2.30 

2.30 

4.90 

0.04 

Error 

22 

10.32 

0.47 



TxP 

1 

0.01 

0.01 

0.01 

0.94 

CM x [T x PI 

1 

0.42 

0.42 

0.57 

0.46 

Error 

22 

16.20 

0.74 



WX x P 

1 

1.17 

1.17 

1.30 

0.27 

CM x [WX x P] 

1 

0.13 

0.13 

0.14 

0.71 

Error 

22 

19.82 

0.90 



T x WX x P 

1 

1.51 

1.51 

2.60 

0.13 

CM x [T x WX x P] 

1 

0.88 

0.88 

1.52 

0.24 

Error 

22 

12.74 

0.58 




♦Bold F values are statistically significant at the p ratio shown in the table. 
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Table D3. Geography Category ANOVA Summary Table 


Source 

df 

SS 

MS 

F 

P 

Between subjects 

Crew member (CM) 

1 

0.26 

0.26 

0.29 

0.60 

Error 

22 

19.20 

0.87 



Within subjects 

Precision (T) 

1 

4.38 

4.38 

4.83 

0.04 

CM x T 

1 

0.05 

0.05 

0.05 

0.83 

Error 

22 

19.95 

0.91 



Weather (WX) 

1 

0.42 

0.42 

0.67 

0.43 

CMxWX 

1 

1.17 

1.17 

1.87 

0.19 

Error 

22 

13.78 

0.63 



Phase (P) 

1 

0.05 

0.05 

0.09 

0.77 

CM xP 

1 

0.63 

0.63 

1.19 

0.29 

Error 

22 

11.70 

0.53 



Tx WX 

1 

1.88 

1.88 

3.99 

0.06 

CM x [T x WX1 

1 

0.13 

0.13 

0.28 

0.61 

Error 

22 

10.37 

0.47 



TxP 

1 

0.01 

0.01 

0.01 

0.93 

CM x [T x P] 

1 

0.42 

0.42 

0.78 

0.39 

Error 

22 

11.95 

0.54 



WX x P 

1 

0.01 

0.01 

0.02 

0.90 

CM x [WX x P] 

1 

0.01 

0.01 

0.02 

0.90 

Error 

22 

6.37 

0.29 



T x WX x P 

1 

0.05 

0.05 

0.16 

0.69 

CM x [T x WX x PI 

1 

0.05 

0.05 

0.16 

0.69 

Error 

22 

6.28 

0.29 




♦Bold F value is statistically significant at the p ratio shown in the table. 




Table D4. Missed Approach Category ANOVA Summary Table 


Source 

df 

SS 

MS 

F 

P 

Between subjects 

Crew member (CM) 

1 

60.75 

60.75 

6.78 

0.02 

Error 

22 

197.00 

8.96 



Within subjects 

Precision (T) 

1 

123.52 

123.52 

29.96 

0.01 

CM x T 

1 

7.52 

7.52 

1.82 

0.20 

Error 

22 

90.71 

4.12 



Weather (WX) 

1 

12.00 

12.00 

4.84 

0.04 

CM x WX 

1 

0.75 

0.75 

0.30 

0.59 

Error 

22 

54.50 

2.48 



Phase (P) 

1 

261.33 

261.33 

27.49 

0.01 

CM x P 

1 

36.75 

36.75 

3.87 

0.07 

Error 

22 

209.17 

9.51 



Tx WX 

1 

50.02 

50.02 

31.40 

0.01 

CM x fT x WX] 

1 

9.19 

9.19 

5.77 

0.03 

Error 

22 

35.04 

1.59 



TxP 

1 

22.69 

22.69 

4.73 

0.05 

CM x [T x P] 

1 

6.02 

6.02 

1.26 

0.28 

Error 

22 

105.54 

4.80 



WX x P 

1 

2.08 

2.08 

0.58 

0.46 

CM x [WX x P] 

1 

0.33 

0.33 

0.09 

0.77 

Error 

22 

79.33 

3.61 



T x WX x P 

1 

46.02 

46.02 

11.88 

0.01 

CM x [T x WX x P] 

1 

2.52 

2.52 

0.65 

0.43 

Error 

22 

85.21 

3.87 




♦Bold F values are statistically significant at the p ratio shown in the table. 
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Table D5. Navigation Category ANOVA Summary Table 


Source 

df 

SS 

MS 

F 

P 

Between subjects 

Crew member (CM) 

1 

10890.19 

10890.19 

9.83 

0.01 

Error 

22 

24360.88 

1107.31 



Within subjects 

Precision (T) 

1 

1064.08 

1064.08 

14.20 

0.01 

CM x T 

1 

234.08 

234.08 

3.12 

0.10 

Error 

22 

1648.08 

74.91 



Weather (WX) 

1 

808.52 

808.52 

9.77 

0.01 

CM x WX 

1 

42.19 

42.19 

0.51 

0.49 

Error 

22 

1821.54 

82.80 



Phase (P) 

1 

623.52 

623.52 

0.94 

0.35 

CM xP 

1 

4740.19 

4740.19 

7.11 

0.02 

Error 

22 

14657.54 

666.25 



Tx WX 

1 

0.75 

0.75 

0.01 

0.93 

CM x [T x WX1 

1 

0.33 

0.33 

0.00 

0.96 

Error 

22 

2053.17 

93.33 



TxP 

1 

14.08 

14.08 

0.16 

0.70 

CM x [T x P] 

1 

36.75 

36.75 

0.42 

0.53 

Error 

22 

1908.42 

86.75 



WX x P 

1 

2.52 

2.52 

0.04 

0.86 

CM x [WX x PI 

1 

315.19 

315.19 

4.55 

0.05 

Error 

22 

1524.54 

69.30 



T x WX x P 

1 

10.08 

10.08 

0.04 

0.84 

CM x [T x WX x P] 

1 

1.33 

1.33 

0.01 

0.94 

Error 

22 

4987.83 

226.72 




♦Bold F values are statistically significant at the p ratio shown in the table. 
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Table D6. Navigation Aids Category ANOVA Summary Table 


Source 

df 

SS 

MS 

F 

P 

Between subjects 


Crew member (CM) 

1 

3906.02 

3906.02 

15.25 

0.01 

Error 

22 

5635.79 

256.17 



Within subjects 


Precision (T) 

1 

12.00 

12.00 

0.74 

0.41 

CM x T 

1 

0.08 

0.08 

0.01 

0.95 

Error 

22 

358.92 

16.31 



Weather (WX) 

1 

4.69 

4.69 

0.10 

0.76 

CMxWX 

1 

46.02 

46.02 

1.01 

0.33 

Error 

22 

1000.29 

45.47 



Phase (P) 

1 

2715.02 

2715.02 

26.52 

0.01 

CM x P 

1 

247.52 

247.52 

2.42 

0.14 

Error 

22 





Tx WX 

1 

1 14.08 

114.08 

7.70 

0.02 

CM x [T x WX] 

1 

44.08 

44.08 

2.98 

0.10 

Error 

22 

325.83 

14.81 



TxP 

1 

0.75 

0.75 

0.03 

0.88 

CM x [T x P] 

1 

3.00 

3.00 

0.10 

0.76 

Error 

22 

652.75 

29.67 



WX x P 

1 

15.19 

15.19 

0.62 

0.44 

CM x [WX x P] 

1 

4.69 

4.69 

0.19 

0.67 

Error 

22 

535.63 

24.35 



Tx WXxP 

1 

40.33 

40.33 

1.57 

0.23 

CM x [T x WX x P] 

1 

8.33 

8.33 

0.32 

0.58 

Error 

r? i n • •<- . . . 

22 

564.83 

25.67 




*Bold F values are statistically significant at the p ratio shown in the table. 




Table D7. Obstructions Category ANOVA Summary Table 


Source 

df 

SS 

MS 

F 

P 

Between subjects 

Crew member (CM) 

1 

627.13 

627.13 

7.83 

0.02 

Error 

22 

1761.24 

80.06 



Within subjects 

Precision (T) 

1 

236.30 

236.30 

12.15 

0.01 

CM x T 

1 

198.05 

198.05 

10.18 

0.01 

Error 

22 

428.03 

19.46 



Weather (WX) 

1 

49.01 

49.01 

5.04 

0.04 

CM x WX 

1 

8.76 

8.76 

0.90 

0.36 

Error 

22 

214.12 

9.73 



Phase (P) 

1 

7.92 

7.92 

0.19 

0.67 

CM x P 

1 

227.51 

227.51 

5.41 

0.03 

Error 

22 

925.45 

42.07 



Tx WX 

1 

2.30 

2.30 

0.15 

0.71 

CM x [T x WX1 

1 

15.76 

15.76 

1.01 

0.33 

Error 

22 

342.82 

15.58 



TxP 

1 

49.01 

49.01 

3.36 

0.09 

CM x [T x P] 

1 

3.80 

3.80 

0.26 

0.62 

Error 

22 

321.07 

14.59 



WX x P n 

1 

97.76 

97.76 

7.59 

0.02 

CM x [WX x PI 

1 

138.38 

138.38 

10.75 

0.01 

Error 

22 

283.24 

12.88 



T x WX x P 

1 

22.01 

22.01 

2.30 

0.15 

CM x [T x WX x PI 

1 

15.76 

15.76 

1.65 

0.22 

Error 

22 

210.61 

9.57 




*Bold F values are statistically significant at the p ratio shown in the table. 
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Table D8. Other Runway Category ANOVA Summary Table 


Source 

df 

SS 

MS 

F 

P 

Between subjects 

Crew member (CM) 

1 

8.76 

8.76 

1.53 

0.23 

Error 

22 

126.12 

5.73 



Within subjects 

Precision (T) 

1 

0.05 

0.05 

0.05 

0.83 

CM xT 

1 

0.26 

0.26 

0.28 

0.61 

Error 

22 

20.32 

0.92 



Weather (WX) 

1 

0.42 

0.42 

0.28 

0.61 

CMxWX 

1 

0.63 

0.63 

0.41 

0.53 

Error 

22 

33.57 

1.53 



Phase (P) 

1 

47.01 

47.01 

23.99 

0.01 

CM x P 

1 

0.01 

0.01 

0.00 

0.96 

Error 

22 

43.12 

1.96 



Tx WX 

1 

1.17 

1.17 

0.83 

0.38 

CM x [T x WX] 

1 

0.42 

0.42 

0.30 

0.60 

Error 

22 

31.03 

1.41 



TxP 

1 

0.05 

0.05 

0.02 

0.89 

CM x fT x P] 

1 

0.26 

0.26 

0.13 

0.73 

Error 

22 

44.82 

2.04 



WX x P 

1 

0.42 

0.42 

0.26 

0.62 

CM x [WX x P] 

1 

0.63 

0.63 

0.38 

0.55 

Error 

22 

36.07 

1.64 



T x WX x P 

1 

0.26 

0.26 

0.16 

0.70 

CM x [T x WX x P] 

1 

1.17 

1.17 

0.72 

0.41 

Error 

22 

35.70 

1.62 




♦Bold F value is statistically significant at the p ratio shown in the table. 
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Table D9. Validation Category ANOVA Summary Table 


Source 

df 

SS 

MS 

F 

P 

Between subjects 

Crew member (CM) 

1 

10.08 

10.08 

0.28 

0.61 

Error 

22 

792.83 

36.04 



Within subjects 

Precision (T) 

1 

0.33 

0.33 

0.05 

0.82 

CM x T 

1 

0.33 

0.33 

0.05 

0.82 

Error 

22 

133.58 

6.07 



Weather (WX) 

1 

3.52 

3.52 

0.56 

0.47 

CMxWX 

1 

0.52 

0.52 

0.08 

0.78 

Error 

22 

138.71 

6.31 



Phase (P) 

1 

3316.69 

3316.69 

10639 

0.01 

CM x P 

1 

42.19 

42.19 

1.35 

0.26 

Error 

22 

685.88 

31.18 



Tx WX 

1 

20.02 

20.02 

4.05 

0.06 

CM x [T x WX1 

1 

0.02 

0.02 

0.00 

0.95 

Error 

22 

108.71 

4.94 



TxP 

1 

1.69 

1.69 

0.27 

0.61 

CM x [T x PI 

1 

0.52 

0.52 

0.08 

0.78 

Error 

22 

136.54 

6.21 



WX x P 

1 

6.75 

6.75 

0.82 

0.38 

CM x [WX x P] 

1 

0.08 

0.08 

0.01 

0.93 

Error 

22 

180.42 

8.20 



T x WX x P 

1 

27.00 

27.00 

6.79 

0.02 

CM x [T x WX x PI 

1 

0.75 

0.75 

0.19 

0.67 

Error 

22 

87.50 

3.98 




*Bold F value is statistically significant at the p ratio shown in the table. 
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Table DIO. Visibility Requirement Category ANOVA Summary Table 


Source 

df 

SS 

MS 

F 

P 

Between subjects 

Crew member (CM) 

1 

14.08 

14.08 

1.82 

0.20 

Error 

22 

170.40 

7.75 



Within subjects 

Precision (T) 

1 

11.02 

11.02 

9.52 

0.01 

CM x T 

1 

0.00 

0.00 

0.00 

1.00 

Error 

22 

25.48 

1.16 



Weather (WX) 

i 

20.02 

20.02 

6.16 

0.03 

CM x WX 

1 

3.00 

3.00 

0.92 

0.35 

Error 

22 

71.48 

3.25 



Phase (P) 

1 

168.75 

168.75 

57.50 

0.01 

CM x P 

1 

1.69 

1.69 

0.58 

0.46 

Error 

22 

64.56 

2.94 



Tx WX 

1 

6.02 

6.02 

2.40 

0.14 

CM x [T x WX] 

1 

0.75 

0.75 

0.30 

0.60 

Error 

22 

55.23 

2.51 



TxP 

1 

10.08 

10.08 

9.09 

0.01 

CM x [T x P] 

1 

0.02 

0.02 

0.02 

0.90 

Error 

22 

24.40 

1.11 



WX x P 

1 

8.33 

8.33 

6.12 

0.03 

CM x [WX x P] 

1 

0.19 

0.19 

0.14 

0.72 

Error 

22 

29.98 

1.36 



T x WX x P 

1 

5.33 

5.33 

4.76 

0.05 

CM x [T x WX x P] 

1 

1.02 

1.02 

0.91 

0.36 

Error 

22 

24.65 

1.12 




♦Bold F values are statistically significant at the p ratio shown in the table. 




Appendix E 

Means and Standard Deviations 

Tables El through E 10 contain the means and standard deviations for the 10 approach chart categories. 


Table El. Means and Standard Deviations for Navigation Categoiy ANOVA 



VOR 

ILS 


Good weather 

Adverse weather 

Good weather 

Adverse weather 

PF: 





Preapproach mean 

23.50 

29.83 

20.42 

25.08 

Preapproach SD 

14.74 

17.69 

12.64 

1633 

Approach mean 

11.50 

11.92 

10.00 

11.25 

Approach SD 

8.59 

6.52 

6.21 

12.72 

PNF: 





Preapproach mean 

31.83 

34.42 

25.58 

27.50 

Preapproach SD 

16.74 

16.95 

16.80 

12.52 

Approach mean 

36.00 

43.58 

28.50 

36.58 

Approach SD 

27.23 

30.17 

18.59 

23.20 


Table E2. Means and Standard Deviations for Missed Approach Category ANOVA 



VOR 

ILS 


Good weather 

Adverse weather 

Good weather 

Adverse weather 

PF: 





Preapproach mean 

3.25 

6.58 

3.67 

1.67 

Preapproach SD 

1.60 

2.78 

2.54 

1.16 

Approach mean 

0.50 

0.83 ' 

0.58 

0.42 

Approach SD 

0.67 

1.12 

0.67 

0.67 

PNF: 





Preapproach mean 

4.17 

6.25 

3.17 

2.58 

Preapproach SD 

2.69 

3.86 

1.64 

1.68 

Approach mean 

3.25 

3.58 

1.42 

2.08 

Approach SD 

2.93 

3.87 

1.51 

1.93 


Table E3. Means and Standard Deviations for Visibility Requirement Category ANOVA 



VOR 

ILS 


Good weather 

Adverse weather 

Good weather 

Adverse weather 

PF: 





Preapproach mean 

1.58 

2.75 

1.08 

1.42 

Preapproach SD 

1.73 

2.14 

0.67 

1.08 

Approach mean 

0.00 

0.08 

0.00 

0.00 

Approach SD 

0.00 

0.29 

0.00 

0.00 

PNF: 





Preapproach mean 

1.75 

4.08 

1.75 

2.17 

Preapproach SD 

1.42 

4.68 

132 

1.90 

Approach mean 

0.17 


0.17 

0.58 

Approach SD 

0.58 


0.58 

1.08 
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Table E4. Means and Standard Deviations for Navigation Aids Category ANOVA 



VOR 

ILS 


Good weather 

Adverse weather 

Good weather 

Adverse weather 

PF: 

Preapproach mean 

12.50 

12.67 

14.25 

12.25 

Preapproach SD 

8.81 

5.16 

9.60 

10.09 

Approach mean 

3.08 

2.75 

3.58 

3.08 

Approach SD 

3.00 

2.42 

2.19 

4.93 

PNF: 

Preapproach mean 

17.50 

21.75 

21.42 

18.00 

Preapproach SD 

6.86 

937 

8.48 

9.14 

Approach mean 

12.33 

15.67 

14.33 

15.33 

Approach SD 

9.44 

11.13 

8.70 

10.42 


Table E5. Means and Standard Deviations for Validation Category ANOVA 



VOR 

ILS 


Good weather 

Adverse weather 

Good weather 

Adverse weather 

PF: 





Preapproach mean 

9.33 

9.92 

10.50 

8.50 

Preapproach SD 

6.07 

3.83 

5.84 

4.44 

Approach mean 

0.08 

0.08 

0.58 

0.50 

Approach SD 

039 

0.29 

1.00 

0.80 

PNF: 





Preapproach mean 

8.67 

9.58 

10.08 

8.00 

Preapproach SD 

4.42 

3.58 

5.11 

439 

Approach mean 

1.67 

1.67 

1.50 

2.00 

Approach SD 

2.74 

137 

1.93 

2.83 


Table E6. Means and Standard Deviations for Obstructions Category ANOVA 



VOR 

ILS 


Good weather 

Adverse weather 

Good weather 

Adverse weather 

PF: 





Preapproach mean 

6.50 

5.58 

5.83 

4.42 

Preapproach SD 

5.02 

5.09 

6.69 

434 

Approach mean 

4.17 

2.92 

5.17 


Approach SD 

2.21 

2.23 

4.57 

2.76 

PNF: 





Preapproach mean 

12.67 

6.92 

5.08 

3.42 

Preapproach SD 

3.73 

5.05 

338 

2.71 

Approach mean 

9.58 

12.58 

7.08 

9.17 

Approach SD 

9.93 

733 

3.68 

5.42 
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Table E7. Means and Standard Deviations for Final Items Category ANOVA 



VOR 

ILS 


Good weather 

Adverse weather 

Good weather 

Adverse weather 

PF: 





Preapproach mean 

0.00 

0.42 

0.67 

0.75 

Preapproach SD 

0.00 

0.67 

0.99 

0.87 

Approach mean 

0.00 

0.00 

0.00 

0.25 

Approach SD 

0.00 

0.00 

0.00 

0.62 

PNF: 





Preapproach mean 

0.42 

0.67 

0.67 

0.92 

Preapproach SD 

1.00 

0.89 

0.99 

1.24 

Approach mean 

0.33 

0.17 

0.33 

0.92 

Approach SD 

1.16 

0.58 

0.89 

1.73 


Table E8. Means and Standard Deviations for Other Runways Category ANOVA 



VOR 

ILS 


Good weather 

Adverse weather 

Good weather 

Adverse weather 

PF: 





Preapproach mean 

1.00 

1.33 

1.08 

0.83 

Preapproach SD 

1.48 

1.72 

1.56 

1.75 

Approach mean 

0.17 

0.00 

0.00 

0.17 

Approach SD 

039 

0.00 

0.00 

0.58 

PNF: 





Preapproach mean 

1.58 

1.33 

1.83 

1.25 

Preapproach SD 

2.47 

1.50 

2.69 

1.77 

Approach mean 

0.33 

0.67 

0.67 

0.33 

Approach SD 

0.65 

1.07 

130 

0.78 


Table E9. Means and Standard Deviations for Communications Category ANOVA 



VOR 

ILS 

— . 


Good weather 

Adverse weather 

Good weather 

Adverse weather 

PF: 





Preapproach mean 

0.42 

0.00 

0.25 

0.00 

Preapproach SD 

1.44 

0.00 



Approach mean 

0.17 

0.25 


0.33 

Approach SD 

0.58 

0.62 


0.89 

PNF: 





Preapproach mean 

0.00 

0.08 

0.08 

0.00 

Preapproach SD 

0.00 

0.29 

0.29 

0.00 

Approach mean 

1.25 

2.17 

1.75 

1.25 

Approach SD 

1.60 

2.76 

2.49 

2.56 
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Table E10. Means and Standard Deviations for Geography Category ANOVA 



VOR 

ILS 


Good weather 

Adverse weather 

Good weather 

Adverse weather 

PF: 





Preapproach mean 

0.00 

0.00 

0.67 

0.17 

Preapproach SD 


0.00 

2.02 

0.58 

Approach mean 

0.00 

0.00 

0.50 

0.00 

Approach SD 


0.00 

0.91 

0.00 

PNF: 





Preapproach mean 

0.00 

0.17 

0.25 

0.25 

Preapproach SD 

0.00 

0.39 

0.62 

0.45 

Approach mean 

0.00 

0.25 

0.58 

0.42 

Approach SD 

0.00 

0.62 

1.51 

0.67 
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Appendix F 
Application 

The results of this study could have widespread 
applications and implications concerning approach infor- 
mation presentation formats. In appendix F the focus is 
on how the managing approach plate information study 
(MAPLIST) findings might be applied to the redesign of 
existing approach charts and to the design of future elec- 
tronic formats. 

The primary finding of this study was that pilots 
appear to mentally organize and use approach chart 
information according to 10 primary categories: Commu- 
nications , Geography , Validation , Obstructions , Naviga- 
tion , Missed Approach , Final Items , Other Runways, 
Visibility Requirement, and Navigation Aids . These cate- 
gories were found to underlie cognitive processes 
invoked by pilots to accomplish their tasks. The way that 
pilots acquired approach chart information reflected their 
use of cognitive processes. 

As noted in the main text of this paper, research has 
shown that presenting information according to the 
user’s mental model improves acquisition efficiency and 
memory retention while reducing user error and results in 
overall improvements in performance with the system. 
Applying the categories that result from MAPLIST to 
presentation formats might improve pilot performance 
when approach chart information is being used by not 
requiring pilots to organize the information to the same 
extent prior to their using it. 

Approach Charts 

One potential application of the MAPLIST findings 
in the area of approach chart design is to organize the 
information’s presentation by using the similarity catego- 
ries. Since existing approach chart formats resemble this 
type of organization already, a simple regrouping of 
some information may fulfill this design goal. When pos- 
sible, the location of the chart where the information is 
currently being presented will be used. In figure FI and 
in the discussion below, a standard (Jeppesen) layout has 
been used in which the similarity categories and their 
information element types have been arranged. 

Validation items have been placed in the upper right 
comer of the approach chart where most of this informa- 
tion already exists. Since pilots are accustomed to using 
the chart starting at the top and progressing to the bot- 
tom, and since pilots use the Validation items prior to all 
others on the approach chart, this section would appear to 
be in a logical position, given this temporal demand. Air- 
port elevation, airport name, changes, chart date, chart 
index number, effective date, geographic name, notes, 


and procedure name are all members of this similarity 
cluster. Again, most of these items already appear in this 
area of existing chart formats. 

Communications items have been placed in the 
upper left corner of the approach chart. The position of 
these items in the upper left quadrant of the chart does 
not seem to fulfill a temporal requirement and could 
probably be placed elsewhere if further analysis proved it 
to be beneficial. This section of the chart was chosen for 
this discussion since current formats use this area for 
Communications information. Members of this group are 
airport departure frequency, approach frequency. Auto- 
matic Terminal Information Service (ATIS) arrival fre- 
quency, ATIS clearance frequency, ground frequency, 
and tower frequency. As with the previously discussed 
category, most of these items are already presented 
together in current formats. The exceptions (i.e., the 
additions) are airport departure frequency and ATIS 
clearance frequency. 

The plan view section of current formats gives the 
map of the area that surrounds and constitutes the air- 
field. Included in this map is the graphical depiction of 
Obstructions and Navigation Aids. For this reason, the 
Geography category’s information should be depicted in 
this section. The Geography category gives additional 
information concerning these data and consists of the 
latitude-longitude grid, latitude-longitude airport very 
high frequency (VHF) omnidirectional range (VOR), 
magnetic variation, and scaling. 

Navigation Aids is the next category listed in the 
plan view section of the chart. Its members are instru- 
ment landing system (ILS) identifier, ILS localizer fre- 
quency, ILS Morse code, nondirectional beacon (NDB) 
frequency, NDB identifier, NDB Morse code, primary 
facility frequency, primary facility Morse code, VOR 
frequency, VOR identifier, VOR Morse code, and VOR 
name. The plan view is a logical place for this informa- 
tion because it can be located next to the graphical depic- 
tion of the aid. Experience and current chart formats have 
shown that some items will need to be duplicated in vari- 
ous places on a chart, and the first such instance in this 
discussion could be the name of the navigation aid. For 
example, the distance measuring equipment (DME) dis- 
tances (in a category to be discussed below) are depicted 
in various sections of an approach chart and are always 
measured from certain Navigation Aids. The navigation 
aid from which distance is measured for a specific 
instance of a DME distance needs to be shown with the 
DME depiction. However, the inclusion of the entire 
Navigation Aids category in the plan view, instead of 
some kind of distribution, is supported by the way the 
pilots view this information. Obviously the clustering 
analysis showed that pilots viewed this information as 
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Communications 


Validation 


Airport departure frequency Airport elevation Effective date 

Approach frequency Airport name Geographic name 

ATIS arrival frequency Changes Notes 

ATIS clearance frequency Chart date Procedure name 

Ground frequency Chart index number 

Tower frequency 



Figure FI. Example use of derived categories for approach chart redesign. 
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similar, but they also stated in the debriefing and showed 
in the acquisition data that a navigation aid, its fre- 
quency, the identifier, and its Morse code are viewed as 
one chunk of information, and should therefore not be 
separated in its depiction. 

All the Obstructions category data have been 
included in the plan view section. This category consists 
of the highest reference point, minimum safe altitude 
(MSA) on approach procedure, MSA minimum altitude, 
MSA reference point, MSA sector radials, and obstruc- 
tions. Since this information gives the location (some- 
times only abstractly) of the obstruction as well as the 
altitude to stay above, this information should be next to 
the geographic information and the graphical depiction 
of the obstructions. 

The Navigation category was the largest and subse- 
quently was further decomposed, as was stated in the text 
of this paper. When looking at the clustering (fig. 9) at 
lower levels, one can see that the information is clustered 
according to whether it was a fix name, distance, altitude, 
or course (heading). For this discussion, information in 
this category that pertains to the horizontal navigation 
(i.e., fix name, distance, and course) will be placed in the 
plan view section and that which pertains to vertical nav- 
igation (i.e., fix name, distance, and altitude) will be 
depicted in the profile section. Since fix names and dis- 
tances are needed for both horizontal and vertical naviga- 
tion (as well as for cross-checking), distance (non- 
distance measuring equipment (DME)), DME distance, 
fix name at final approach fix (FAF), and fix name are 
duplicated in figure FI (in the plan and profile views). 
This duplication will not necessarily be the same piece of 
information, but it sometimes will be (e.g, fix name at 
FAF). The other information in this category that may be 
presented in the plan view section exclusively is feeder 
route radial, fix name at initial approach fix (IAF), 
inbound course, instrument landing system (ILS) local- 
izer magnetic course, lead radial, outbound course, and 
the procedure turn distance limit. 

Those items in the Navigation category that will also 
appear in the profile section were discussed above (i.e., 
distance (non-DME), DME distance, fix name at FAF, 
and fix name). Those items that pertain exclusively to 
vertical navigation in this category are decision altitude 
(AGL), decision altitude (MSL), FAF intercept (AGL), 
FAF intercept (MSL), glideslope (GS) intercept (AGL), 
GS intercept (MSL), procedure turn altitude, step-down 
altitude (AGL), step-down altitude (MSL), and visual 
descent point. One exception to this breakdown may be 
that of procedure turn altitude. Most of the items listed in 
this paragraph refer to the descent profile near the touch- 
down portion of the approach, and the procedure turn 


may be out further and therefore treated like the MSA on 
approach procedure (i.e., in the plan view). The tight 
clustering of this item with procedure turn distance limit 
(fig. 9) might also suggest its removal from this section 
(and its inclusion in the plan view). 

The other category that is depicted in the profile 
view of figure FI is Final Items. These items are GS 
angle, GS position on target runway, rate of descent, 
touchdown zone elevation (TDZE) target runway, and 
threshold-crossing height. One addition to this category 
is runway length. This item clustered with the other run- 
ways category, but it was felt that it belonged more with 
the Final Items information (refer to discussion in the 
main text). 

The Missed Approach category information has all 
been listed in the section traditionally used for the missed 
approach instruction. Since the pilots view this informa- 
tion as similar, it may be more efficient to present it 
together in one location, but this does not mean that some 
of the information could not be duplicated elsewhere. For 
example, it would probably be useful to depict the 
holding-pattern course in the plan view section for cross- 
checking purposes. It might also be useful to present the 
location of the missed approach point (MAP) in both the 
plan view and profile view for cross checking. However, 
when the pilots are dealing with the missed approach, 
they also may benefit if all of the information is located 
together. 

Visibility Requirement has been placed in figure FI 
according to the position of the chart where it is normally 
presented. However, given the importance pilots placed 
on this information and its frequent acquisition, it might 
be beneficial to place it on the chart in an area that is 
scanned frequently (to promote easy access). Decision 
altitude is also usually depicted where the visibility 
requirement is shown in figure FI. Its inclusion in the 
profile section makes sense considering that the AGL 
altitudes are calculated from the TDZE. However, the 
decision altitude and the visibility requirement are also 
referenced independently to determine whether the pilots 
are still legal to fly the approach and to determine if and 
when to fly a missed approach. For this reason, a separate 
(and maybe duplicate) depiction of this information 
might be warranted. 

The Other Runways category contains the informa- 
tion that is commonly shown on the airport diagram. This 
information consists of GS position on parallel runway, 
other runway numbers, and TDZE on the parallel run- 
way. There was not much activity in this experiment that 
concerned these data, so their position on the chart might 
warrant an infrequently scanned location. 
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Electronic Formats 

How the categories that were found in this study 
might influence electronic presentation formats is similar 
to the ones that were discussed previously for the 
approach chart. Information that is similar should be 
presented together. What differs from the above discus- 
sion is the integration of approach information with simi- 
lar information that is presented already on existing 
displays, instead of in a dedicated display for the elec- 
tronic chart. 

The flight deck used for this discussion will have a 
primary flight display (PFD), navigation display (NAV), 
and an electronic library system (ELS) with its corre- 
sponding display. In addition, it is assumed that a mini- 
mum of two additional displays exist (e.g., engines 
display and caution and warning-checklist display). No 
assumptions that concern the location and size of these 
displays are being made. There is also no attempt here to 
determine or suggest specific formats for the information 
that is presented (e.g., font size, icon). Instead, this dis- 
cussion will be limited to where the information might be 
placed. 

The grouping of information will be the same as that 
discussed previously (i.e., with two Navigation catego- 
ries and with runway length being included with Final 
Items); therefore, only the names of the categories will 
be used in this discussion. 

Presenting this information electronically will auto- 
mate some of the pilots’ current tasks (e.g., getting the 
correct chart), but the validation information is still 
needed for pilots to ascertain what the automation is 
doing and if it is doing it correctly. This type of auxiliary 
information probably would be most appropriate on the 
ELS. Also of that nature, and also targeted for the ELS in 
this discussion is the communications information. 10 

All the information listed above for the plan view 
section of the approach chart could be presented on the 
NAV. These categories were Geography, Navigation 
Aids, Obstructions, and Navigation (1). Many of these 
items are being depicted on the NAV already and only 
need minor modifications to include the subtle differ- 


10 Items destined for the ELS in this discussion will be those that 
have no similar items already presented on glass flight decks with 
which to integrate. 


ences. Other information in these categories may not be 
necessary at all for an advanced flight deck. For example, 
advanced radios could automatically tune and identify 
themselves, thus removing the need for a visual presenta- 
tion of Morse code and possibly even the frequency. 
(Possibly the frequencies could be moved to the ELS for 
validation purposes.) 

The vertical navigation items pose the most diffi- 
culty when they are integrated with existing glass cockpit 
layouts. This information could be added to the NAV 
display with the use of “altitude arcs” and other similar 
plan view aids used for vertical navigation. However, 
given the difference in functions that the Navigation (2) 
and Final Items information support when compared 
with that traditionally on the NAV, it would probably be 
better to dedicate a new display for this purpose. Work 
has been done in the past for vertical navigation displays, 
with many formats resembling the profile section of an 
approach chart with an aircraft icon that indicates the air- 
craft’ s current position relative to the path. This display 
would be needed only during the end of the flight and 
could possibly share duties with an existing screen on the 
flight deck. 

Missed Approach information could be presented on 
both the NAV in graphical form and on the ELS in tex- 
tual form. Another possibility is to present the missed 
approach information on a third screen in the event of a 
missed approach execution. In this case, the missed 
approach courses and MAP information would be pre- 
sented on the NAV, the textual missed approach infor- 
mation would be presented on the ELS, and in the event 
that a missed approach is executed, the appropriate infor- 
mation could be treated as caution and warning informa- 
tion and displayed on the appropriate caution and 
warning screen. Of course with these types of advanced 
display technologies, the possibilities are numerous. For 
example, in the event of a missed approach, the missed 
approach courses and MAP information could also be 
highlighted on the NAV. 

The Visibility Requirement and Other Runways cat- 
egories, probably would be most appropriate on the ELS. 
This information is not often integrated with the other 
information functionally and should probably be treated 
as auxiliary information, even though it is important and 
in the case of Visibility Requirement information often 
accessed. 
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Figure 1 . Approach chart 1 . 
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Figure 3. Approach chart 3. 
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Figure 4. Approach chart 4. 





L-94-00883 

Figure 5. DC-9 simulator and MAPLIST configuration. 
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Figure 6. MAPLIST air traffic control station. 
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Figure 7. Pilot with MAPLIST approach chart interface. 
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Figure 9. Dendogram of similarity ratings. 
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Figure 10. Element clusters with interpretation. 
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Figure 13. Total acquisitions by phase. 












Figure 14. Pilot-flying (PF) acquisitions by phase. 
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Figure 15. Pilot-not- flying (PNF) acquisitions by phase 
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Figure 17. Bad weather acquisitions by phase. 






71 


Figure 18. Instrument landing system (ILS) acquisitions by phase. 
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Figure 20. Category acquisitions by phase. 
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